CoreLibs
CoreLibs copied to clipboard
AdguardTeam
Service worker's cache issue
Issue description
There are several websites that use Service workers' based caching and this messes with AdGuard filtering.
The main issue is that when the page is cached by a service worker there will be no network request for AdGuard to filter. The problem that in this case when AdGuard filters are changed there's no way to forcibly clear the cache and make the browser request updated page version.
How to fix this
We suggest the following solution:
- CoreLibs inject a so-called "service worker content script" into the service worker itself.
- This service worker script should periodically check the time of the last protection restart.
- If the timestamp is different from the previous check (or if it cannot be retrieved), the script should clear the cache.
Still reproducible for me but that tends to just happen. For some reason at random filtering just fails on Outlook while it works on all other sites.
Might try a clean reinstall of both Chrome and AdGuard for Mac nightly and see if it works then.
Did clean installs of both Google Chrome and AG for Mac nightly, and still no Assistant or ad blocking on outlook.live.com however, the site is using AG's certificate. It's just not blocking anything in Chrome. Of course, Firefox works.
And there's nothing really in the browser's console either and no mention of the Assistant. Very strange.
An update, did a clean cache of Safari and it's not working there anymore either - no Assistant or ad blocking. Will do the same to Firefox.
EDIT: And it's working again in Safari and has always worked in Firefox. Hmmm. Also have noticed filtering can fail sometimes on msn.com too. Verrrrrry strange.
Also you can try blocking service worker and see how the behavior changes: ||outlook.live.com/owa/serviceworkerhandler.ashx
No go on either one. Still fails.
EDIT: And now it's failing in Firefox too.
This keeps getting stranger. Sometimes it fails in Chrome, but once in awhile after any amount of random reloads, filtering + Assistant works again... until I reload the page (or clear site data). I don't think the browser matters either, nor Outlook per-say. It seems filtering (and Assistant disappearing) randomly fails in general in AG for Mac nightly. Maybe it's silently crashing (without exiting the app) or something?
I might've figured this one out finally. You're right Andrey, it's probably the service worker. Right now I'm testing this filter and it seems to be working 100% of the time...
||outlook.live.com/mail/sw.js$important
You guys might want to test this one for a little bit, if it doesn't break anything (as far as I can tell, it doesn't) you might consider adding it to the base filter.
The only thing that I can't understand is how can service worker be causing missing ads.
The thing is that service worker caches the page content (which should've been filtered by AG - so no ads).
And like that, even with the service worker blocked, it still randomly fails. There's probably a bug somewhere... I wonder, would AG for Mac nightly have the same filter issue AG for Android nightly has? Maybe it's that, and after filter updates filter rules aren't applying correctly.
EDIT: I might've found a way to help reproduce it on-demand.
I found that if I delete RPSSecAuth portion of the live.com cookie in Chrome (while Outlook is closed) and then open Outlook in a new tab, filtering (cosmetic filtering - e.g. the Annoyances filter has a rule to remove the ad placeholder on the right side) isn't working and the Assistant will disappear. If I close that Outlook tab, and open a new Outlook tab filtering+Assistant works again.
I tested it in Windows with AG for Windows too, however filtering+Assistant works there even if I delete the RPSSecAuth part of the live.com cookie, which is to be expected.
Hopefully that helps reproducing it finally. If not, would debug-level logs help?
Another odd way to reproduce it 100% of the time with the latest AG for Mac nightly + Chrome...
With Chrome open (and no Outlook tab open) check for filter updates in AG for Mac and after the filters are updated (I used the AG Base + Spyware + Annoyances filters in this test, and they were all updated) then open Outlook in a tab. First load there's no ad blocking (e.g. the right ad placeholder appears) and no Assistant. If you refresh the Outlook tab, it works normally.
This is the Outlook URL I use via bookmark (in case that matters): https://outlook.live.com/mail/#/inbox
This seems to only work when there's filter updates. If I check for filter updates and there aren't any, I can't reproduce the issue. That's kinda why I was wondering it was related to the filter issues with AG for Android as it's just plain odd that updating filters causes it to happen.
@BooBerry could you please reproduce the issue and then check Chrome dev tools: https://uploads.adguard.com/up04_igurd_Example_Domain.png
Could it be that the service worker somehow slipped through while AG was reconfiguring protection?
Yeah, I've been watching that when reproducing it, and no, the service worker doesn't load at all according to the dev tools.
It's probably not related to the service worker afterall. Very odd indeed.
@sfionov @szolin guys, check this out:
- Outlook caches the page content using a service worker (which is absolutely okay).
- Alongside the page, it caches links to
local.adguard.org/*/contentScript.js - After AG restart, that link becomes invalid due to the expired auth token.
We should figure what to do with it.
Useful info about service workers:
- https://blog.angular-university.io/service-workers/
- https://www.w3.org/TR/service-workers-1/#extended-http-headers
This may be helpful https://jakearchibald.com/2014/offline-cookbook/#network-only
Guys, again, the first thing we need to address is that cached links to content-script do not work due to expired auth token.
Just to let the Dev's know, a version of this problem is still happening in the nightly released tonight 6-3-2019. See here https://forum.adguard.com/index.php?threads/adguard-assistant-disappears-and-adguard-stops-filtering.31136/ @BooBerry's user filter suggestion seems to fix for me on Gmail
Here's an update.
There's no good generic solution to this that we can implement on the network level.
Options are:
- Suppressing service workers (not a very good solution)
- Handle this with a browser extension. Once we have a native browser assistant extension, we'll be able to handle this properly.
First option could be a stealth mode option, I suppose (for those who want to block them in general)? But yeah, not a good solution at all.
The second option is probably the way to go, IMO.