ipfs
What to do about Manifest V3
As flagged by @dignifiedquire the proposed webextention api changes under the banner of the v3 manifest work will reduce the scope of what ipfs-companion is able to do to bring ipfs to the browser
Proposed changes: https://docs.google.com/document/d/1nPu6Wy4LWR66EFLeYInl3NzzhHzc-qnk4w4PX-0XMw8/edit?usp=sharing
Tracking issue: https://bugs.chromium.org/p/chromium/issues/detail?id=896897&desc=2
Most significant is the move away from the webRequest api to the declarativeNetRequest api https://developer.chrome.com/extensions/declarativeNetRequest
The webRequest api is blocking. Chromium asks the webextention what to do about each outbound http request at runtime. IPFS Companion makes use of this to let the user toggle between redirecting ipfs addresses to a local daemon or the public gateway.
The declarativeNetRequest requires extentions to tell chromium in advance, what urls patterns they wish to block or redirect by defining them statically as rules in the manifest.json
There may well be other important changed for us in that doc, but this one stands out as the most significant.
Oh wow, a lot to unpack here.
After reading the draft with proposed changes I wrote down some highlights with interesting changes and potential pain points (below).
Will follow V3 as it unfolds, but it is very likely it will bring us additional maintenance cost.
Restrict the blocking capabilities of the webRequest API
In Manifest V3, this API will be discouraged (and likely limited) in its blocking form. The non-blocking implementation of the
webRequestAPI, which allows extensions to observe network requests, but not modify, redirect, or block them (and thus doesn't prevent Chrome from continuing to process the request) will not be discouraged. As an alternative, we plan to provide adeclarativeNetRequestAPI . The details of what limitations we may put in thewebRequestAPI are to be determined. [..]
- While deciding whether a request is to be blocked or redirected, the
declarativeNetRequestAPI is given priority over thewebRequestAPI because it allows for synchronous interception.- Evaluation of
declarativeNetRequestrulesets is performed during theonBeforeRequeststage of a network request.
We use blocking capability of webRequest API to inspect headers, detect IPFS resources, do DNSLink lookups and other things – src/lib/ipfs-companion.js#L95-L97:
browser.webRequest.onBeforeSendHeaders.addListener(onBeforeSendHeaders, { urls: ['<all_urls>'] }, ['blocking', 'requestHeaders'])
browser.webRequest.onBeforeRequest.addListener(onBeforeRequest, { urls: ['<all_urls>'] }, ['blocking'])
browser.webRequest.onHeadersReceived.addListener(onHeadersReceived, { urls: ['<all_urls>'] }, ['blocking', 'responseHeaders'])
Personally I don't think this change will stick in the form proposed in the draft as having declarativeNetRequest as the only blocking option makes it impossible to write things like uBlock Origin or uMatrix, which could create terrible PR for Google, given their Ad business.
If the worst comes to the worst, for the basic functionality we may be okay with declarativeNetRequest and some elaborate rules for redirecting https?://*/ip(f|n)s/* and https?://*.ip(f|n)s.*/, but supporting DNSLink in current seamless form may be difficult/impossible (or would require user action/opt-in per site).
Replace background pages with ServiceWorkers
We plan to replace background pages with ServiceWorkers, registered at install time and granted access to extension APIs. [..] We will be able to reduce the amount of extension-specific code since, instead of implementing a custom background-running presence, we can extend the existing ServiceWorker context. Most importantly, the platform will evolve with the open web; as improvements to ServiceWorkers and the web platform are made, the extensions platform benefits from those same improvements.
We need a stable, persistent background presence to be able to run embedded js-ipfs node. It does not sound like SeviceWorkers, which is often killed by the browser when idle. I hope there will be an inexpensive and reliable way to keep SW alive in WebExtension context.
If not, @Gozala had some ideas on keeping ServiceWorker alive in https://github.com/ipfs/in-web-browsers/issues/137#issuecomment-450530548, but the net result will be to actually use more CPU by artificially keeping it alive than when having a background page.
Restricting Origin Access
In Manifest V3, host permissions will be granted by the user at runtime (similar to activeTab, but with options for the user to choose to always run on a specific site or all sites), rather than install-time.
In Manifest V3, we want activeTab-style host permissions to be the default, with a number of extra options. Instead of being granted access to all URLs on installation, extensions will be unable to request
<all_urls>, and instead the user can choose to invoke the extension on certain websites, like they would withactiveTab. Additional settings will be available to the user post-installation, to allow them to tweak behavior if they so desire.
[..] In the default case (click-to-run), it is clear to the user when the extension is running, and has a safe default (not running on any site). When the user chooses to invoke the extension on a given site, there is implicit understanding that the extension will "see" the contents of the page.
This sounds bad, as it suggests by default it will be click-to-run, but in later section about host permissions support for <all_urls> is described as open question:
Open question: Should
activeTabbe removed in favor of specifying <all_urls> in the host_permissions key of the manifest?
Also, this blogpost from 2018 suggests how the UI will look like:
Beginning in Chrome 70, users will have the choice to restrict extension host access to a custom list of sites, or to configure extensions to require a click to gain access to the current page.
Cross-Origin Communication
Beginning in Manifest V3, content scripts will not be given special privileges regarding the requests they can make. If a content script needs access to data fetched from a cross-origin server that requires authentication, it can proxy the request through its background page using extension messaging.
As long they limit this restriction to Content Scripts, we should be fine.
We do CORS-but-not-really to IPFS API only from the background page. "Not really" because we remove Origin header from requests made to API URL by js-ipfs-http-client running in WebExtension context to remove need for manual whitelisting via Access-Control-Allow-Origin at go-ipfs: src/lib/ipfs-request.js#L119-L151.
Promise-Based APIs
Extension APIs will be promise-based. The older callback version will continue to be supported.
I was about to say its cool, as there will be no need for mozilla/webextension-polyfill, but we will probably need a polyfill for all the other breaking changes. Time will tell, so far issues to follow:
- Migrate apps/extension APIs to use Promises instead of callbacks
- Extension APIs should be also exposed under
browser.*to match the WebExtensions spec
Dynamic Content Scripts (P2)
we can allow extensions to dynamically add and remove, or enable and disable, content scripts. This would allow extensions to only add these scripts once they have permission to do so.
There are additional situations in which this is beneficial, as well. Currently, the advice for extensions wishing to dynamically inject scripts based on some knowledge at runtime is to use the
tabs.executeScript API; however, this is insufficient for certain use cases. In particular, this cannot (reliably) insert a script before the page finishes loading, which is a feature that content scripts provide. Allowing dynamic content scripts would solve this use case.
This is good news, as it would solve issue with window.ipfs described in https://github.com/ipfs-shipyard/ipfs-companion/issues/362#issuecomment-362231167
(browser.contentScripts works in Firefox, but it's not implemented in Chrome yet)
Refactoring of various APIs
Additional work and duplicated code, but nothing critical.
Sad realization: even if Firefox follows these V3 API changes, for some time it won't, while Chromium will, so it will be like writing websites in 90s with similar level of code duplication for achieving the same thing in two different browsers :|
ps. what a perfect issue number :ok_hand:
If not, @Gozala had some ideas on keeping ServiceWorker alive in ipfs/in-web-browsers#137 (comment), but the net result will be to actually use more CPU by artificially keeping it alive than when having a background page.
Note that keeping SW alive is going to be an arms race with browsers. Also was proposing for Safari (As replacement for SharedWorker due to lack of it) is different as you'll have a clients (documents served from it) which is considered as legitimate case.
Personally I don't think this change will stick in the form proposed in the draft as having
declarativeNetRequestas the only blocking option makes it impossible to write things like uBlock Origin or uMatrix, which could create terrible PR for Google, given their Ad business.
Not so sure about it, they might ship their own content blocker (that would likely don't harm their Ad business) and call it a day https://bugs.chromium.org/p/chromium/issues/detail?id=896897&desc=2#c23
I'm not sure if there are some way to engage with the corresponding people driving these changes but I think best bet would be to convince them to expose SharedWorker (or equivalent) for long lived tasks or maybe an official API that would provide a way to keep ServiceWorker alive.
Extensions Developer Advocate at Chrome is doing office hours! I grabbed the very first free slot (April 12, 16:30 UTC).
My plan is to give an overview of what ipfs-companion does, potential API improvements and our concerns around Manifest V3.
Update: We gave a brief demo and an overview of our needs during the office hours.
There is still a lot of unknowns around Manifest V3 (especially no details on if/how Service-Worker-like API is going to replace JS process running in persistent background pages).
As for webRequest, activeTab and <all_urls> Privacy Badger team has written up some relevant requests for changes: Chrome team requests-Privacy Badger.pdf
We need to wait until another draft is published to see if our concerns are adressed.
not sure if it's related to this but looks like it by the APIs involved:
on chromium Version 71.0.3578.98 (Developer Build) (64-bit) the extension breaks on activation with the attached log from the background script: nibjojkomfdiaoajekhjakgkdhaomnch-1555915381867.log
Some updates/clarifications from Simeon (Extensions Developer Advocate): https://groups.google.com/a/chromium.org/d/msg/chromium-extensions/veJy9uAwS00/9iKaX5giAQAJ
[.. ] I want to emphasize that the Manifest V3 design document is not exhaustive or immutable. The extensions team is pursuing the goals outlined in this design document and iterating on design and implementation details. The best way for developers to really understand the changes is to experiment with the Manifest V3 platform.
[..] the extensions team is currently working on a Developer Preview of Manifest V3. Our goal with this preview is to give developers a way to start experimenting with some of the most significant changes to the platform in order to provide targeted feedback.
When they ship Developer Preview, we will prototype a port and see how V3 impacts Companion in practice, but based on analisys from Raymond Hill (uBlockOrigin) I would not be too optimistic:
The blocking ability of the webRequest API is still deprecated, and Google Chrome's limited matching algorithm will be the only one possible, and with limits dictated by Google employees
It's annoying that they keep saying "the webRequest API is not deprecated" as if developers have been worried about this -- and as if they want to drown the real issue in a fabricated one nobody made. – https://github.com/uBlockOrigin/uBlock-issues/issues/338#issuecomment-496009417
Response from the Ghostery team suggests the manifest v3 performance claim does not hold. https://whotracks.me/blog/adblockers_performance_study.html tl;dr degradation is milliseconds at worst
Update from Google https://blog.chromium.org/2019/06/web-request-and-declarative-net-request.html
Chrome provides enterprise controls through its administrator policies. The blocking version of the Web Request API remains available for managed extensions because of the deep integrations that enterprises may have between their software suites and Chrome.
Pretty bad news, means current onBeforeRequest it won't work out of the box and we need to re-implement entire redirect logic using "The Declarative Net Request API"
The Declarative Net Request API now allows for the registration and removal of dynamic rules - specified at runtime rather than statically in the manifest. We’ve also added the capability to remove common tracking headers, such as Referer, Cookie, and Set-Cookie.
In our case the devil is in the details.
Setting up a crude redirect of /ipfs/{cid} paths with this API should work fine. When changing gateway Companion would remove/add new redirect rule for /ipfs/* paths.
Problem: we lose ability to do CID validation, so we need to work round false-positives.
We could do DNSLink check in non-blocking onBeforeRequest and add redirect rules based on that (similar to current best-effort strategy, but it will continue to yield false-negatives on initial request.
We may lose ipfs:// handler done via search-query hack, so would be great if real handler landed before that happens:
- https://bugs.chromium.org/p/chromium/issues/detail?id=651311
- https://bugs.chromium.org/p/chromium/issues/detail?id=64100#c12
Shall see when Developer Preview lands.
Related discussion thread: https://groups.google.com/a/chromium.org/forum/#!msg/chromium-extensions/qFNF3KqNd2E/8R9PWdCbBgAJ
A number of extension developers have reached out to ask how Mozilla plans to respond to the changes proposed in v3. Following are answers to some of the frequently asked questions.
ipfs-companion/issues/808: Permanent in-depth review on Chrome Web Store:
Google makes it more and more difficult to publish powerful extensions such as IPFS Companion.
In preparation for Manifest v3 (https://github.com/ipfs-shipyard/ipfs-companion/issues/666) Chrome Web Store artificially slows down publishing of extensions that requesti access to certain APIs.
Chrome team rolled out Manifest V3 into Canary: https://groups.google.com/a/chromium.org/forum/#!msg/chromium-extensions/hG6ymUx7NoQ/TiAe0gI5AQAJ
In order to help developers start experimenting with and transitioning to MV3 we've put together a migration guide. This doc provides a high-level look at what's changing and what developers will need to do to adapt. We also have a guide specifically for migrating from background pages to service workers.
I will dig into this deeper next week, but from a quick read it seems we need to refactor most of Companion to support both V2 (Brave, Firefox) and V3 (Google Chrome, future Chromium). It is unclear if we will be able to persist an embedded js-ipfs node in V3 at all.
Quick API Checklist vs Impact on IPFS Companion (:ballot_box_with_check: means YES)
- :ballot_box_with_check: Do you have host permissions in your permissions or optional_permissions arrays in manifest.json?
- :ballot_box_with_check: Are you using background pages?
- :ballot_box_with_check: Are you using the browser_action or page_action property in manifest.json?
- :ballot_box_with_check: Are you using the chrome.browserAction or chrome.pageAction JavaScript API?
- :ballot_box_with_check: Are you currently using the blocking version of chrome.webRequest in a consumer-facing extension?
- :ballot_box_with_check: Are you currently using
chrome.tabs.executeScript({code: '...'}),eval(), ornew Function()in background contexts or content scripts? - :ballot_box_with_check: Are you currently using chrome.runtime.getBackgroundPage(),
- :man_shrugging: Are you making CORS requests in content scripts?
- :ballot_box_with_check: Are you using a custom content_security_policy in manifest.json?
Redirects with V3 are broken at the moment:
- Google published migration guide
- Microsoft Edge is preparing for Manifest V3 as well.
Manifest V3 now seems inevitable. After we wrap ongoing work in Q4, should build a PoC.
Adblocker Dev Summit 2020
- Some history of ad blocking and limitations of declarative filtering approach on Apple platforms: https://www.youtube.com/watch?v=_dduavvDj6s
- Status of Manifest v3 in Firefox: https://www.youtube.com/watch?v=tpDFS-GUytg
- Status update on Chrome side: https://www.youtube.com/watch?v=zxWG2yUcTFA
Originally posted in https://github.com/uBlockOrigin/uBlock-issues/issues/338#issuecomment-725727365
Google, Microsoft, Apple, and Mozilla have launched the WebExtensions Community Group (WECG) to collaborate on standardizing browser extensions:
- https://www.bleepingcomputer.com/news/security/google-microsoft-and-mozilla-work-together-on-better-browser-extensions/
- https://github.com/w3c/webextensions/blob/main/charter.md
Working towards removing annoying differences between vendors is a good news, it also means the direction of "Manifest V3" is no longer controlled by Google on their own, and Microsoft's experiments with PWA and protocol handlers have some overlap with our use cases.
Thin on details, time will tell if we end up in better place.
It has begun: https://developer.chrome.com/docs/extensions/mv3/intro/mv3-migration/
Mozilla will implement it as well: https://blog.mozilla.org/addons/2021/05/27/manifest-v3-update/
As of this writing, we are hoping to complete enough work on this project to support developer testing in Q4 2021 and start accepting v3 submissions in early 2022.
Over the past week or so I've been familiarizing myself with the current ipfs-companion extension and the limits of the manifest v3 spec.
I've put together a repo of very basic prototypes of local node redirects with DeclarativeNetRequest and ServiceWorker. Most of the pros and cons have been documented above but I've included the most glaring from this experiment below.
👉 Repo https://github.com/meandavejustice/ipfs-mv3-prototype
General Manifest v3 Cons
- <all_urls> is no longer supported
DeclarativeNetRequest
Pros:
- no service worker weirdness
- The approved replacement for blocking WebRequest 👍
Cons:
- subdomain forwarding is harder (I haven't gotten this to work yet)
- cannot verify URLs
- cannot verify CIDs
- cannot check DNSLink (may be able to do this in a separate service worker)
ServiceWorker
Pros:
- can check urls and cids against isIPFS package
- should be able to make a non blocking webRequest to check DNSLink
Cons:
- Currently using a sketchy keepAlive method in the service worker, performance is questionable along with whether or not this will pass review
I had a misunderstanding around the removal of <all_urls>. We can still access the urls we need with "https://*/" and "http://*/" being added to host_permissions key.
This makes me feel much better about our options here. I will add a prototype of adding new rules with updateDynamicRules api to https://github.com/meandavejustice/ipfs-mv3-prototype shortly. (this is essentially what @lidel describes above in https://github.com/ipfs/ipfs-companion/issues/666#issuecomment-502072347)
Opinion piece from EFF: https://www.eff.org/deeplinks/2021/11/manifest-v3-open-web-politics-sheeps-clothing
I've added a branch to ipfs-mv3-prototype to support dnslink domains.
Steps used are:
- confirm that the url is not an IPFS url
- call
/api/v0/dnsto confirm the domain has a dnslink mapping. - add a dynamic rule using the
updateDynamicRulesmethod.
When running the extension you'll see a console message with the rule id, domain, and cid. On next load of the domain it will be redirected to local gateway.
This exposed our limitation of making requests to the IPFS API, we need to have a way to update the Access-Control-Allow-Origin headers. We used to do this with the onBeforeSendHeaders listener, but we've lost that with the webRequest blocking api. This is being tracked in #1036
For the record, two years later (https://github.com/ipfs/ipfs-companion/issues/666#issuecomment-590852186), redirects and UI updates based on navigation with Manifest V3 are still effectively broken in Chromium:
- TLDR: iiuc we are unable to use
chrome.tabs.onUpdated.addListenerorchrome.webRequest.onCompleted.addListenerreliably because stopped service worker (background page) does not get wake up correctly every time.- Details: Issue 1024211: Manifest V3: webRequest listeners not called after service worker stops
- :clown_face: :point_right: Google stopped accepting new V2 extensions this January 2022
- This means one can't create new extensions that rely on this functionality: old API V2 is gone, new one V3 is broken
@lidel Re
service worker (background page) does not get wake up correctly every time.
See https://github.com/guest271314/persistent-serviceworker
i18n.getMessage not supported in mv3 extensions
tracking => https://bugs.chromium.org/p/chromium/issues/detail?id=1268098
EDIT: Looks like it was fixed at end of January, just hasn't showed up in stable yet. Will verify in chromium latest
https://bugs.chromium.org/p/chromium/issues/detail?id=1268098#c13
Filed issue with chromium tracker for matching ipfs:// protocol urls with the declarativeNetRequest api https://bugs.chromium.org/p/chromium/issues/detail?id=1306869
I thought I had filed this issue last week, turns out I didn't so here is the new one => https://bugs.chromium.org/p/chromium/issues/detail?id=1309190
Above is an issue documenting running into the limits of regexFilter in declarativeNetRequest
Bugzilla meta issue for MV3 last updated 9 months ago
I've had some trouble logging into Mozilla's replacement for IRC, https://wiki.mozilla.org/Matrix, I've reached out to friend who's at mozilla to try and get some info on MV3 status.
Manifest v3 in Firefox: Recap & Next Steps (https://blog.mozilla.org/addons/2022/05/18/manifest-v3-in-firefox-recap-next-steps/):
-
Mozilla will maintain support for blocking WebRequest in MV3. To maximize compatibility with other browsers, we will also ship support for declarativeNetRequest.
-
we have decided to support Event Pages in MV3, and our developer preview will not include Service Workers (we’re continuing to work on supporting these for a future release).
-
Work is continuing in parallel, and we expect to launch MV3 support for all users by the end of 2022.