:chrome-extension target support

[thheller]

WIP: everything here might change

master currently has support for a basic proof of concept for building Chrome Extensions with shadow-cljs.

The idea is to take a manifest.edn which is basically the standard manifest.json but in EDN + some extra special properties that will be used to configure the compiler.

For background scripts an :shadow/entry namespaces is configured

 :background
 {:shadow/entry demo.chrome-bg
  :persistent false}

This will be read by the :target and then generate the following .json for dev builds

 "background":
 {"persistent":false,
  "scripts":
  ["out/background.js", "out/cljs-runtime/goog.debug.error.js",
   "out/cljs-runtime/goog.dom.nodetype.js",
   "out/cljs-runtime/goog.string.string.js",
   "out/cljs-runtime/goog.asserts.asserts.js",
   "out/cljs-runtime/goog.reflect.reflect.js",
   "out/cljs-runtime/goog.math.long.js",
   "out/cljs-runtime/goog.math.integer.js",
   "out/cljs-runtime/goog.object.object.js",
   "out/cljs-runtime/goog.array.array.js",
   "out/cljs-runtime/goog.structs.structs.js",
   "out/cljs-runtime/goog.functions.functions.js",
   "out/cljs-runtime/goog.math.math.js",
   "out/cljs-runtime/goog.iter.iter.js",
   "out/cljs-runtime/goog.structs.map.js",
   "out/cljs-runtime/goog.uri.utils.js",
   "out/cljs-runtime/goog.uri.uri.js",
   "out/cljs-runtime/goog.string.stringbuffer.js",
   "out/cljs-runtime/cljs.core.js",
   "out/cljs-runtime/clojure.string.js",
   "out/cljs-runtime/shadow.cljs.devtools.client.console.js",
   "out/cljs-runtime/shadow.module.shared.append.js",
   "out/cljs-runtime/demo.chrome_bg.js",
   "out/cljs-runtime/shadow.module.background.append.js"]},

It will add the "scripts" which chrome will load in order. For release builds these will be compacted down and optimized to a single file.

:content-scripts function the same. An :entry is specified and it generates the "js" file references. Currently the assumption of only one content script is hardcoded but could easily support multiple scripts.

Basically everything that wants to use some kind of JS will gain a special key and the compiler should generate the appropriate code. Since EDN can express everything JSON can express and more we can easily "enhance" the config and automatically modify it based on needs.

Currently the minimal config required is

  :chrome-ext
  {:target :chrome-extension
   :extension-dir "out/chrome-ext"}

It assumes that the :extension-dir contains a manifest.edn (eg. out/chrome-ext/manifest.edn). It will generate all files into an out directory which is currently hardcoded but could become configurable.

Code is generated via :modules meaning that everything uses the same code and things are only compiled once. This is far superior to having a separate build for each.

Not sure if this is viable for all build scenarios but so far it looks promising.

One known issue regarding the REPL highlights a long standing problem about dealing with multiple clients. Since everything will connect to the same REPL endpoint the server-side REPL doesn't know what to eval in. So there needs to be a way to "switch" the target.

[pepe]

Not sure, if it is relevant, but have you seen https://github.com/binaryage/chromex?

[thheller]

@pepe I have seen it but it is not relevant. It is just a wrapper for the API (which you can use if you want) but it doesn't do anything related to building/packaging.

[thheller]

I just pushed a pretty significant rewrite of the internals for this and the config now works differently.

Given this manifest.edn

{:name "Getting Started Example"
 :version "1.0"
 :description "Build an Extension!"
 :manifest-version 2

 :shadow/outputs
 {:inject
  {:output-type :chrome/single-file
   :init-fn demo.chrome.manual-inject/init}

  :browser-action
  {:init-fn demo.chrome.browser-action/init}

  :content-script
  {:init-fn demo.chrome.content/init
   :chrome/options {:matches ["http://localhost:*/*"]
                    :run-at "document_idle"}}

  :background
  {:init-fn demo.chrome.bg/init}}

 :browser-action
 {:default-title "hello world"
  :default-icon "icon.png"
  :default-popup "browser-action.html"}

 :content-security-policy
 ["default-src 'self';"
  ;; FIXME: unsafe-eval should be injected for dev, user shouldn't have to write this
  "script-src 'self' 'unsafe-eval' http://localhost:9630;"
  "connect-src * data: blob: filesystem:;"
  "style-src 'self' data: chrome-extension-resource: 'unsafe-inline';"
  "img-src 'self' data: chrome-extension-resource:;"
  "frame-src 'self' data: chrome-extension-resource:;"
  "font-src 'self' data: chrome-extension-resource:;"
  "media-src * data: blob: filesystem:;"]}

The new thing is :shadow/outputs which controls all the outputs generated by shadow-cljs. Each is basically a :module (see :browser) target which have an implicit shared dependency. This is just for code-splitting reason and reducing the amount of actual code generated.

The above config will generate 4 files in the :extension-dir configured in the shadow-cljs.edn config which is unchanged.

:chrome-ext
{:target :chrome-extension
 :extension-dir "out/chrome-ext"}

:shadow/ouputs could be moved here instead and I'm still debating whether this would be a better place to have them or the manifest.edn. Feedback welcome.

Either way the 4 files generated will be

• out/background.js and a "background" entry will be added to the manifest.json linking all the scripts required to load the background code. You should never manually reference out/background.js and instead rely on it being injected into the manifest.

 "background":
 {"scripts":
  [ ... lots of files ...],
  "persistent":false}}

• out/content-script.js and a "content_scripts" entry will be added to the manifest.json with the extra :chrome/options. Same as out/background.js. Never reference this manually anywhere.

 "content_scripts":
 [{"js":
   [ ... lots of files ...],
   "matches":["http://localhost:*/*"],
   "run_at":"document_idle"}],

• out/browser-action.js. Directly usable from the browser-action.html via <script src="out/browser-action.js"></script>

• out/inject.js. I have not tested this yet but the intent of this is being able to have a single file so you can run chrome.tabs.executeScript(tabId, {file: "out/inject.js", ...) to inject a content-script programmatically. It might be best to use a separate build for this so I'm not too sure about keeping this.

[thheller]

Oops, another thing I added is the support for :init-fn demo.chrome.bg/init. This function will be called when your module is loaded. If you don't want that you can use the usual :entries [your.ns] just as :browser. The intent of this is to make it easier to have something that executes once and doesn't run each time your ns is reloaded via live-reload.

[bbss]

:shadow/ouputs could be moved here instead and I'm still debating whether this would be a better place to have them or the manifest.edn. Feedback welcome.

I did think it was a bit odd that the manifest includes shadow concerns. That said I am quite new to shadow-cljs and maybe it makes sense.

[thheller]

Yeah the first iteration co-located the config entries within the chrome config but was limiting in some ways. Now it could be moved since its just the :shadow/outputs key. Could even make manifest.edn optional and just rewrite a standard manifest.json.

[thheller]

2.3.31 will automatically re-configure (and compile) the build when the manifest.edn config file is changed.

The build config now also supports setting :manifest-file which should be a path to a manifest.edn input file you want to use. The output will always be generated into :extension-dir + manifest.json.

  :chrome-ext
  {:target :chrome-extension
   :extension-dir "out/chrome-ext"
   :manifest-file "out/chrome-manifest.edn"}

This makes it possible to set a different manifest for dev and release, e.g. :release {:manifest-file "..."}

You may also now put the :shadow/outputs config into the build config instead under the :outputs key since we don't need to namespace here.

[miguelsm]

This is great! thanks so much for adding support for web extensions.

Is there a reason to force persistent to be false. That setting is not compatible with the webRequest API: https://developer.chrome.com/extensions/background_pages#persistentWarning. Can it be set back to true somehow?

Also it'd be nice if shadow-cljs output scripts were appended to any pre-existing arrays set in the background scripts or content-script js properties. That way it would be possible to include some external scripts like the webextensions-polyfill which I haven't managed to require in my .cljs files from either the distributable js file nor the npm module.

[thheller]

You should be able to set :background {:persistent true} in your manifest.edn to override the default.

I'll see about adding extra scripts for the polyfills. You might be able to get them to work by just requiring and setting :js-options {:language-out :ecmascript6 }.

[roosta]

Hello, thank you for adding web extension support to shadow-cljs. I've been converting a Firefox extension to shadow-cljs and the experience have been excellent save for when trying to compile a release build:

IllegalArgumentException: No method in multimethod 'flush-optimized-module' for dispatch value: :chrome/browser-action
        clojure.lang.MultiFn.getFn (MultiFn.java:156)
        clojure.lang.MultiFn.invoke (MultiFn.java:233)
        shadow.build.output/flush-optimized/fn--11775 (output.clj:373)
        shadow.build.output/flush-optimized (output.clj:369)
        shadow.build.output/flush-optimized (output.clj:352)
        shadow.build.targets.chrome-extension/process (chrome_extension.clj:284)
        shadow.build.targets.chrome-extension/process (chrome_extension.clj:271)
        clojure.lang.Var.invoke (Var.java:381)

Removing :browser-action from :shadow/outputs allows the remainder to compile. I'm very new to shadow-cljs and I easily could be missing something.

[thheller]

Yeah I need to revisit the whole :chrome-extension thing. This will be a whole lot easier now that :loader-mode :eval exists. Should take away a lot of pain points.

I'll see about adding a quick fix for release however. You can probably use this for now:

:shadow/outputs
{:foo
  {:output-type :chrome/single-file
   :init-fn demo.chrome.browser-action/init}}

and then use foo.js in the browser action .html page. There are just a few bits missing for the automated :browser-action optimizations.

[roosta]

That works. Thanks!

[isker]

Hi! Sorry to nag, but I'm wondering whether there are plans to move this out of experimental status. Thanks!

[thheller]

@isker the main issue is the unwritten documentation. Otherwise it seems solid enough but I currently don't have enough time to do the final polish and write the documentation so it'll probably be a while still.

[ro6]

@thheller Do you have the beginning of some documentation written, or does it need to be started from scratch? I'm working on an extension now and might be able to start documenting as I figure things out. Is the source that generates the user manual available somewhere to contribute to?

[thheller]

@ro6 I didn't start anything yet no. Docs source is https://github.com/shadow-cljs/shadow-cljs.github.io, contribs are very welcome.

[ro6]

@thheller Would you prefer I just throw things I notice in here as I go since this is experimental, or open new issues for everything?

It looks like the machinery for calling reload callbacks (eg marked with :^dev/before-load) doesn't distinguish between the different JS environments of the extension, so in the page console (the conent_script context) I'm seeing:

shadow.cljs.devtools.client.browser.js:49 shadow-cljs: can't find fn background/stop
shadow.cljs.devtools.client.browser.js:49 shadow-cljs: load JS content_script.cljs
cljs.core.js:165 :content-script/loaded
shadow.cljs.devtools.client.browser.js:49 shadow-cljs: can't find fn background/start

Check out the repro here.

[ro6]

@thheller Is there support for opening a different REPL to each of the JS environments? Based on what I'm seeing right now, evaluations are sent to the background page. I'd like to be able to evaluate in the content_script context as well.

[thheller]

shadow-cljs supports selecting the JS runtime yes. Unfortunately no tool supports that so it isn't very accessible right now.

About the other issue please open a new one.

[yqrashawn]

@thheller commented on Nov 3, 2019, 5:37 PM GMT+8:

shadow-cljs supports selecting the JS runtime yes. Unfortunately no tool supports that so it isn't very accessible right now.

About the other issue please open a new one.

Any updates on this? I'm building a pretty large extension involving multiple runtimes. I found shadow/repl-runtime-select, but can't make it work. I have to constantly close tabs to make the repl connect to the desired runtime.

[ro6]

@thheller commented on Nov 3, 2019, 5:37 PM GMT+8:

shadow-cljs supports selecting the JS runtime yes. Unfortunately no tool supports that so it isn't very accessible right now. About the other issue please open a new one.

Any updates on this? I'm building a pretty large extension involving multiple runtimes. I found shadow/repl-runtime-select, but can't make it work. I have to constantly close tabs to make the repl connect to the desired runtime.

@yqrashawn I'm in the same boat. I think shadow/repl-runtime-select worked for a while, but not anymore.

I've been pondering the ergonomics. What UX do you imagine? Swapping a single nREPL session between runtimes?

It would be cool to annotate at the file/ns or form levels to specify one or more "eval target(s)".

[thheller]

See https://github.com/thheller/shadow-cljs/issues/776

I'll see if I can get that back to working shape this weekend.

[thheller]

Closing this because I think :chrome-extension is no longer needed. V3 extensions can use ESM just fine and that should be the way forwarnd. See https://github.com/thheller/shadow-cljs/issues/1051.