Manifest v3 background scripts should not be killed when there are active listeners

[ParticleCore]

My extension requires a very important API to be always active in the background script in order to intercept and modify web requests as early as possible, and during the entire session of that page.

• This extension is simply built to work on www.youtube.com
• It requires modifying certain aspects of the page, and files it loads, as early as possible and whenever new versions are loaded since they can expire. To achieve this, the background script makes use of the browser.webRequest.onBeforeRequest API
• YouTube is not a typical website in which each page is loaded as a brand new page, instead it loads only once and then modifies the page contents as the user navigates, or interacts with the page, so one typical page session can go on for hours
• YouTube often updates their script files, which means that while the user is navigating for a long time some of those files can expire and new ones are loaded.
• For this reason the extension must be always active in order to intercept newer file versions that are loaded way past the first 30s of when the page was initially opened.

With the new v3 implementation of non-persistent background scripts, the extension is no longer capable of working at all past the first 30s window, at which time users that navigate or interact with the page will no longer have anything related to the extension working correctly.

As a result of this, I am forced to revert the manifest back to v2 and I sincerely hope that this issue can be resolved, otherwise it will be the death of this and many other extensions in the same predicament.

Why can't we keep background scripts alive while there are active listeners is beyond me, most of those permissions can only be used by background scripts too so even if we wanted to migrate them to content-scripts we just can't. At the very least, provide the option for the user to authorize an extension to run permanently on the background since it is all moving towards user-based permission models.

[wolfspelz]

We moved to manifest 3, but it

• cost 20 k€ to develop,
• made the code more complex,
• the service is worse sometimes,
• now costs more resources (network and cpu) to offer the same service to the user as before (it's a messenger. Messengers need persistence).
  But no one seeems to listen. As if some management decided and now they go for it, because... it's been decided. Sticking to this although the necessary changes to extensions needing persistence completely defeat the purpose of less background resource use. This change either
• does nothing,
• is circumvented by malicious extensions or
• is counterproductive because you can compensate for missing persistence with more cpu and traffic.
  (I am not angry, I am just disappointed)

[ParticleCore]

@wolfspelz If possible to inform, were you able to keep background persistence for your messenger by migrating it into a dedicated service worker script with a timed messaging between ports to keep it alive over using the now limited background scripts?

[dotproto]

Speaking as a browser contributor, the additional complexity associated with an event-based background context is not lost on us. The problem is that browsers cannot guarantee a long lived execution context in the same way that we used to, especially as WebExtensions are being brought to mobile devices.

While I’m not formally affiliated with Mozilla, I am currently contributing to Firefox WebExtensions and addons.mozilla.org. I was involved in the recent launch of extensions on Firefox for Android, and before this I spent a few years on Chrome. Mobile OSs and devices have made strides over the past decade, but they are still significantly more resource constrained than typical desktop environments. More specifically, Firefox for Android cannot guarantee that an extension background context will stay alive due to Android‘s process model. I expect Safari on iOS has similar constraints.

While consensus opinion among browser vendors seems to be that we must adopt an event-based background context in WebExtensions, we are sympathetic to the challenges this poses for developers. Over the past year I believe all participating browser engines have tweaked how extension lifetimes are managed in order to allow background contexts to stay alive longer. For example, both Firefox and Chrome have adopted the behavior that extension API calls will extend the lifetime of the background context. But since we can’t guarantee that the process an extension is running in will stay alive, developers still need to program defensively and guard against unexpected termination.

With respect to the assertion that your MV3 extension “now costs more resources (network and cpu) to offer the same service to the user as before”, any hard data you can share here would be welcome. This potentially includes code, extension packages, your test suite, metrics recorded, etc.

[ParticleCore]

@dotproto I believe you might have replied here in respect to @wolfspelz reply, not the original issue I opened.

But on the topic you discussed, if I am understanding you correctly is that the foundation for the decision to put lots of extensions on the death row is simply to benefit Android browsers mainly? Which I cannot describe just how unconscionable that is given that many extensions (mine included) don't even target, nor plan targeting mobile platforms due to how different each website can behave, look, interface between both platforms, leading to even bigger challenges and unmanageable complexity for small dev teams or even single devs.

Sure, it might not be feasible to ensure long background sessions on mobile, but why are we still ignoring the PC platform? Why is is such an impossible task to consider allowing the user to decide to allow persistent background scripts on PC? It's not mobile, and it's not witchcraft, because we already have it in v2. It makes no sense at all this agonizing effort to destroy so many good extensions.

Who exactly are these changes benefiting in the end?

The users? Because they will have less extensions and a bit more battery?

The developers? Who won't be able to keep their extensions working anymore?

In the end it always sounds like it benefits just one group, and none of them are the target audience nor the creative audience, and it is a bleak future if this does not change.

[Pauan]

@dotproto It doesn't make any sense to restrict all extensions (including desktop extensions) because of limitations on mobile. Mobile limitations should only affect mobile, not desktop.

Firefox allows for persistent extensions on desktop, Chrome does not.

This is not a technical decision, because Chrome is perfectly capable of supporting persistent extensions on desktop (and indeed they did support them for many years).

It is purely a dogmatic decision.

[wolfspelz]

Restricting WebExtensions for the benefit of mobile devices in Chrome is kind of funny as Chrome on Android does not support extension. Sound so absurd that I suppose the error lies with me and I am getting something wrong(?)

[newRoland]

Listening to webRequest.onBeforeRequest should wake the background up when needed. Why does the background have to be active all the time?

[Pauan]

@newRoland This has been extremely well documented in other threads. But the short version is: some extensions need to keep data in memory in order to function properly, or the extension needs to do expensive computations which last more than 5 minutes. Automatically killing the background page breaks those extensions.

[dotproto]

I wish I could address the concerns you've all raised here in more detail, but I don't have the time or brain power right now. For the moment, I'll simply share my abbreviated (and therefore likely more incendiary than I intend) perspective.

@dotproto I believe you might have replied here in respect to @wolfspelz reply, not the original issue I opened.

Correct, my last comment was in response to @wolfspelz's comment.

But on the topic you discussed, if I am understanding you correctly is that the foundation for the decision to put lots of extensions on the death row is simply to benefit Android browsers mainly?

No, as I see it the foundation for this fundamental change to the way browser extensions work is in service of the end user's experience, the reach of the WebExtensions platform, and it's sustainability.

Sure, it might not be feasible to ensure long background sessions on mobile, but why are we still ignoring the PC platform? Why is is such an impossible task to consider allowing the user to decide to allow persistent background scripts on PC?

It doesn't make any sense to restrict all extensions (including desktop extensions) because of limitations on mobile. Mobile limitations should only affect mobile, not desktop.

Firefox allows for persistent extensions on desktop, Chrome does not.

IMO WebExtensions, much like the web platform on which they're based, should be write once run anywhere. In order to achieve this, there must be a common set of capabilities that work everywhere. An event based background context provides this, whether in the form of a service worker or event page.

Browsers are not ignoring the PC platform. Changes have been made across browsers that allow event-based background contexts to stay alive indefinitely. There are no more hard limits on maximum execution time. Extensions should be able to continue running as long as they're doing something every 30 seconds. Depending on your extension's design, this may require you to jump through hoops to provide that signal, but it is nonetheless possible to do. For example, about 6 months ago Chrome added documentation on how to use WebSockets in service workers, including the use of a keepAlive() function.

As I see it, the fact this is a bit awkward to do is a feature, not a bug. Most extensions should not expect to stay alive forever and the browser cannot guarantee that behavior if you do write your extension with that expectation, even on desktop. (Though admittedly unexpected termination on desktop is much rarer.) Should the platform provide a better way to do this? Perhaps. And that's something I expect we'll revisit this year.

Who exactly are these changes benefiting in the end?

Users. A common web extensions platform allows them to expect that when they install an extension on one device, it will work on any other. The experience they have should adapt to where they're using it. Their browser's sync features should ensure they have all of their tools available regardless of where they log in. They should be able to experience the web as they want everywhere.

[Pauan]

@dotproto There are no more hard limits on maximum execution time. Extensions should be able to continue running as long as they're doing something every 30 seconds.

And as has been extensively documented in other threads, that does not support every use case for every extension. There are extensions which break with that model, because they require persistence.

Depending on your extension's design, this may require you to jump through hoops to provide that signal, but it is nonetheless possible to do.

And those hoops were not necessary in the past. We simply marked our extensions as persistent in the manifest.json and all was right in the world.

The fact that we now have to do hacks like constantly polling a timer just to keep our background page alive is... ridiculous.

It's wasting resources for no good reason, making extensions slower, not faster. A config option consumes less resources than constantly polling a timer.

Just make it so that transient extensions are the default, but extensions can opt-in to being persistent with a config option (or perhaps a permission). This problem is incredibly easy to fix, and yet it has not been fixed in years.

You seem to be trying to force developers to make their extensions transient, but that will not work, because many extensions do require persistence, they cannot be changed.

It's not because of extension developer stubbornness or laziness, persistence is fundamentally necessary for those extensions. They cannot be made transient, no matter how hard you push for it.

So you are excluding those extensions from the ecosystem, which is to the detriment of the users who wish to use those extensions.

[dotproto]

Apologies for the delay, @ParticleCore. The title of this issue specifically asks that the background context be kept alive “when there are active listeners”. Depending on what you mean, this may already be supported.

If by "active listener" you mean an extension event listener that receives events more than once every 30 seconds, then I believe this is already covered. To my knowledge, both Firefox and Chrome will currently extend the background context’s 30 second idle timer when a new extension event is received. I believe both browsers also support extending the lifetime when in extension API is called. I’m less confident about Safari simply because I haven’t discussed this issue directly with engineers on that team.

On the other hand, if by "active listener" you mean an extension that has any event listener registered, I’m not sure that’s the right signal to extend a background context's lifetime. There are many cases where an extension may subscribe to an infrequent event and only wished to be woken when that event fires.

[ParticleCore]

No, as I see it the foundation for this fundamental change to the way browser extensions work is in service of the end user's experience, the reach of the WebExtensions platform, and it's sustainability.

That's what keeps being repeated around, but the actual reality is a different thing.

This justification ignores the fact that not even most websites themselves provide the same pages to desktop and mobile platforms. This point alone is a big reason why many ext devs (like me) refuse to develop an extension for mobile.

Mind you, for many of us the biggest barrier in developing for mobile is not the difference in programming for two different extension environments, but the fact that the pages themselves are completely different in both layout and behavior. If we wanted to develop for Android, we actually did it despite these issues.

It is a complete mistake to think that making it easier to develop for both platforms will result in better offering for mobile and not consider how much it will ruin the end users' experience on desktop.

This entire ordeal looks like a huge decision was made with little to no actual regard and thought put behind it, and no consideration for the downsides, and all of it hidden behind a single fragile justification that has yet to be proven worthwhile when faced with the significant number of compromises.

IMO WebExtensions, much like the web platform on which they're based, should be write once run anywhere. In order to achieve this, there must be a common set of capabilities that work everywhere.

That entire argument falls flat when we consider Firefox iOS, and we have exceptions for webRequest already, the feature is already fragmented from Chrome's implementation, so it won't write once and run anywhere.

Extensions should be able to continue running as long as they're doing something every 30 seconds. Depending on your extension's design, this may require you to jump through hoops to provide that signal, but it is nonetheless possible to do. For example, about 6 months ago Chrome added documentation on how to use WebSockets in service workers, including the use of a keepAlive() function.

And isn't this a direct contradiction of the original argument in favor of it? Exactly what difference is there between this and providing a "persistent" flag to the background worker? I have yet to understand what is precisely the arguments being made against this very simple solution to everyone's problem.

• If the dev wants the background work to persist, he can implement a 30s looping signal => the script stays running forever in the background ?
• Or he could just enable a persistent flag in the manifest and not worry about timers => the script stays running forever in the background too ?
• Regardless of being mobile or desktop, both will do the exact same thing, and yet only one is not only accepted, but recommended too?

Nothing about this makes any sense, the entire discussion right now comes down to "we could just let you enable a single flag and make life a lot easier for you like it was before, but we would prefer for you to jump through hoops and make your life worse" all in "service of the end user's experience"?

Again, to me that does not make any sense. The dev experience is being made much worse for no valid reason.

Should the platform provide a better way to do this? Perhaps. And that's something I expect we'll revisit this year.

I do to, hopefully it will be a good improvement.

Users. A common web extensions platform allows them to expect that when they install an extension on one device, it will work on any other.

Again, it completely ignores the impossible unification due to a serious misconception of anything that desktop does, mobile does the same.

That discussion keeps grouping 2 very distinct types of extensions, and it should never do that; extensions that depend on the website content, and extensions that do not depend.

The later is way easier to make it work since it only focuses on the capabilities of the browser and not much else, while the former will depend on the websites being served with the exact same content on both platforms, or, if not, then will be forced to support two different versions of their extension (mobile vs desktop), and that's assuming the devs will want to do that.

Because if they don't then this change mostly benefits a subset of the extensions, while the others pay the price.

I will keep trying to make my extension work with V3, probably going to try migrating the background script to the service worker model to see if the timed loop is enough a workaround for my problem, but I am definitely not confident about this unless Firefox provides an official documentation stating that any service worker with running code every 30s or less will not be killed, otherwise it will be another one of those unintended behaviors they eventually patch and we are left with nothing.

In any case, it definitely sounds like we should stop using background scripts and just move to service workers to avoid any headaches, I just wish that was properly announced.

[ParticleCore]

If my active listener, you mean an extension event listener that receives events more than once every 30 seconds, then I believe this is already covered. To my knowledge, both Firefox and Chrome will currently extend the background context’s 30 second idle timer when a new extension event is received. I believe both browsers also support extending the lifetime when in extension API is called. I’m less confident about Safari simply because I haven’t discussed this issue directly with engineers on that team.

Seems like the timed loop is only applicable to service workers, not background scripts. I am unsure if this is an issue for me at this time, I will try to migrate to the service worker to see if it is enough of a work around for my scenario.

However, I cannot for the life of me agree that having a background script persistent via a manifest flag is somehow using up more resources, and energy than the exact same script running a timed loop constantly. It's just not reasonable to even entertain that thought while at the same time advocating for it on the basis of better performance and energy consumption for the users.

But then again, I have not yet tested or profiled such scenarios so I would only be saying this as purely my own unverified opinion, but would love to see an official test showing the difference between the two methods.