shadow.js.shim.module$vega_embed is not a function

[kovasap]

I'm using oz in my CLJS project and have been getting shadow.js.shim.module$vega_embed is not a function errors when I try to render plots with oz.core/vega-lite ever since switching to webpack. It seems to me that during the switch something about the vega_embed dependency is not being properly installed.

To my untrained eyes, it seem like everything is set up properly though:

public/js/cljs-runtime/shadow.js.shim.module$vega_embed.js:

goog.provide("shadow.js.shim.module$vega_embed");
goog.provide("module$shadow_js_shim_module$vega_embed");
shadow.js.shim.module$vega_embed = shadow$bridge("vega-embed");
module$shadow_js_shim_module$vega_embed.default = shadow.js.shim.module$vega_embed;

//# sourceMappingURL=shadow.js.shim.module$vega_embed.js.map

package.json:

{
  "devDependencies": {
    "shadow-cljs": ">=2.16.12",
    "webpack-cli": "^4.9.2"
  },
  "dependencies": {
    "bulma": "^0.9.3",
    "csv-parse": "^5.0.4",
    "csv-stringify": "^6.0.5",
    "highlight.js": "11.1.0",
    "react": "^17.0.2",
    "react-data-grid": "7.0.0-beta.11",
    "react-dom": "^17.0.2",
    "stream-browserify": "^3.0.0",
    "vega-embed": "^6.20.7"
  }
}

target/index.js:

// WARNING: DO NOT EDIT!
// THIS FILE WAS GENERATED BY SHADOW-CLJS AND WILL BE OVERWRITTEN!

var ALL = {};
ALL["csv-parse/sync"] = require("csv-parse/sync");
ALL["csv-stringify/sync"] = require("csv-stringify/sync");
ALL["vega"] = require("vega");
ALL["react-dom"] = require("react-dom");
ALL["vega-embed"] = require("vega-embed");
ALL["react"] = require("react");
ALL["react-data-grid"] = require("react-data-grid");
global.shadow$bridge = function shadow$bridge(name) {
  var ret = ALL[name];

  if (ret === undefined) {
     throw new Error("Dependency: " + name + " not provided by external JS. Do you maybe need a recompile?");
  }

  return ret;
};

shadow$bridge.ALL = ALL;

Any ideas what might be causing this issue?

[thheller]

Did you run webpack AFTER shadow-cljs has finished compiling? It generates the target/index.js so you need to run webpack every time this file may have changed. The only time it should change is when adding new npm dependencies. They are also never hot reloaded, so adding a new npm dependency will require reloading the entire page. Adding new npm requires in the REPL is also not supported since they are provided by webpack which will only apply on page load.

[kovasap]

I ran webpack and shadow-cljs multiple times one after the other when debugging this issue - unfortunately, whichever order even multiple times doesn't seem to resolve this issue :(.

[thheller]

And you are loading the generated webpack output before loading the CLJS output? (ie. script tags in the correct order and not async)

Also the error message implies that you are doing (:require ["vega-embed" :as x]) and then calling x as a function. That may have changed if you are not using the vega-embed version that oz is expected. Did you verify you are using the correct version?

[kovasap]

Yep, my webpack output is being loaded first (and I'm not having issues with any other library):

    <script src="/js/libs.js/main.js" type="text/javascript"></script>
    <script src="/js/app.js" type="text/javascript"></script>

I'm actually calling through oz only, which must itself contain the (:require ["vega-embed" :as x]) statement. When I run webpack after running shadow-cljs without vega-embed installed, it errors out saying:

Module not found: Error: Can't resolve 'vega-embed' in '/home/kovas/biomarker-correlator/target'
resolve 'vega-embed' in '/home/kovas/biomarker-correlator/target'

After running npm i vega-embed this error goes away. Running my code with vega-embed 6.20.5 (in my package.json) worked before switching to webpack, and now does not. So I don't think the version is the problem.

[kovasap]

I think i might see the problem. In the vega-embed documentation, they say to use import embed from 'vega-embed'; when running from a javascript file. But in the oz code the import is (:require ["vega-embed" :as vegaEmbed]) and the call is (vegaEmbed ...). Calling the function as vegaEmbed is what the vega-embed docs say to do when you are running with javascript embedded in your code (and you are importing via a script tag). This explains why I started seeing this issue after switching to webpack.

It seems like this problem could be fixed by changing the oz code to use the shadow-cljs :default import pattern. However, I'm not sure if that would work for other users of the library.

Ideally at this point I could introduce some workaround to my code that would let me use the oz codebase as-is (perhaps some kind of alias in my shadow-cljs or webpack config?). Or maybe there is another way to sort this out for my use case?

I'm a bit worried that I'll run into this issue again if I try to use another cljs library with webpack!

[thheller]

This is not so much a problem with webpack. shadow-cljs will have the same issues. The JS/npm world is transitioning from commonjs to ESM which means what they export may change. Default exports didn't exist in commonjs and don't translatte 100% cleanly. So if libs indeed switch to ESM and also switch what they export it needs to change on the side they are imported too.

That is just the situation and there is no easy workarround if that is the case.

The :default pattern is old and deprecated. The newer $default mentioned in the official blogpost should be used instead. Still need to update the docs properly.

[kovasap]

Hmm ok, are you suggesting that the only way to solve this problem then is to change oz? If so i'm kinda surprised given that it worked for me when I wasn't using webpack.

[kovasap]

I'm still unsure about how to proceed with this issue, or what direction to go in to debug. Any ideas?

[thheller]

I really cannot answer without a reproducible example. With or without webpack, just need something to actually look at.

[kovasap]

Here you go! https://github.com/kovasap/oz-webpack-repro

[thheller]

Thanks for the repro. As expected webpack does indeed default to using the "module" build, which means that the old commonjs style const vegaEmbed = require("vega-embed") becomes import vegaEmbed from "vega-embed". On the surface that may look similar but indeed it is not. See the translation examples in the User's Guide.

shadow-cljs does not yet use "module" by default yet but will have to at some point. Which will break in the same way. In fact you can get there today by setting :js-options {:entry-keys ["module" "browser" "main"]} in your build config. This is unfortunate but something the JS world decided to do.

So at some point oz will have to switch (:require ["vega-embed" :as vegaEmbed]) to (:require ["vega-embed$default" :as vegaEmbed]). This transition is annoying but not something we can avoid.

[kovasap]

Ok, based on my understanding then the logical next step is to send a PR to oz changing this require. Would that change break any other ways of importing and using oz?

And just to double check, there's no way to work around this without changing oz?

On Sat, Feb 19, 2022, 10:51 PM Thomas Heller @.***> wrote:

Thanks for the repro. As expected webpack does indeed default to using the "module" build, which means that the old commonjs style const vegaEmbed = require("vega-embed") becomes import vegaEmbed from "vega-embed". On the surface that may look similar but indeed it is not. See the translation examples in the User's Guide.

shadow-cljs does not yet use "module" by default yet but will have to at some point. Which will break in the same way. In fact you can get there today by setting :js-options {:entry-keys ["module" "browser" "main"]} in your build config. This is unfortunate but something the JS world decided to do.

So at some point oz will have to switch (:require ["vega-embed" :as vegaEmbed]) to (:require ["vega-embed$default" :as vegaEmbed]). This transition is annoying but not something we can avoid.

— Reply to this email directly, view it on GitHub https://github.com/thheller/shadow-cljs/issues/988#issuecomment-1046175204, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACC3NARVLHSE64RAIRCNRVTU4CFNRANCNFSM5OJ32SZQ . You are receiving this because you authored the thread.Message ID: @.***>

[thheller]

Changing the require on oz side will break everything still using the CommonJS variant (eg. default shadow-cljs). Definitely can't do that yet.

I guess oz could add a check on what it calls. For example

(def vegaEmbed* (if (fn? vegaEmbed) vegaEmbed vegaEmbed/default))

and then use vegaEmbed* instead of vegaEmbed later.

[kovasap]

Ok cool, I'll send a PR their way! One question: does shadow-cljs have a way to depend on code that is not published as a clojar (https://shadow-cljs.github.io/docs/UsersGuide.html#_clojure_script)? For instance something like this: https://github.com/reifyhealth/lein-git-down. I want to be sure the PR solves my problem before submitting.

[thheller]

You can use deps.edn or project.clj. shadow-cljs.edn itself does not support git dependencies.

[kovasap]

Ok cool, verified that this fixes the issue with deps.edn. PR has been sent!

On Sun, Feb 20, 2022 at 3:10 PM Thomas Heller @.***> wrote:

You can use deps.edn or project.clj. shadow-cljs.edn itself does not support git dependencies.

— Reply to this email directly, view it on GitHub https://github.com/thheller/shadow-cljs/issues/988#issuecomment-1046341860, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACC3NAXBGQEFN57TTUSRBFLU4FYFFANCNFSM5OJ32SZQ . You are receiving this because you authored the thread.Message ID: @.***>