Make it possible to link runtime JavaScript file together with OCaml libraries

[vouillon]

Quick hack to make it possible to include some JavaScript files when compiling an OCaml library.

For now, one has to pass the --linkall and --no-runtime flags:

js_of_ocaml --linkall --no-runtime runtime.js library.cma

At the moment, the generated code uses some ECMAScript 6 features.

Trying to address https://github.com/ocsigen/js_of_ocaml/issues/1508

[gpetiot]

I've tried out this branch and I still get runtime errors when loading the js files:

I have modified the command line invokation of jsoo like this: https://github.com/ocaml-doc/voodoo/pull/114/commits/43bdc26986265d53481268a9cd394d75400c39da I checked quickly the result of the build and it looks like it exports what needs to be exported:

Object.assign
  (globalThis.jsoo_runtime,
   {caml_make_local_vect: caml_make_local_vect,
    Base_unsafe_create_local_bytes: Base_unsafe_create_local_bytes,
    caml_csel_value: caml_csel_value,
    Base_am_testing: Base_am_testing,
    Base_hash_double: Base_hash_double,
    Base_hash_string: Base_hash_string,
    Base_int_math_int64_pow_stub: Base_int_math_int64_pow_stub,
    Base_int_math_int_pow_stub: Base_int_math_int_pow_stub,
    Base_int_math_int64_ctz: Base_int_math_int64_ctz,
    Base_int_math_nativeint_ctz: Base_int_math_nativeint_ctz,
    Base_int_math_int_ctz: Base_int_math_int_ctz,
    Base_int_math_int32_ctz: Base_int_math_int32_ctz,
    Base_int_math_int64_clz: Base_int_math_int64_clz,
    Base_int_math_nativeint_clz: Base_int_math_nativeint_clz,
    Base_int_math_int_clz: Base_int_math_int_clz,
    Base_int_math_int32_clz: Base_int_math_int32_clz,
    Base_caml_exn_is_most_recent_exn: Base_caml_exn_is_most_recent_exn,
    Base_clear_caml_backtrace_pos: Base_clear_caml_backtrace_pos,
    Base_int_math_int_popcount: Base_int_math_int_popcount});

So does that mean everything is okay up to here and the error is caused by the toplevel? For the package Base that I use to check the PoC, I load the following cma's:

"stdlib/caml.cma.js";
"stdlib/md5_lib.cma.js";
"stdlib/shadow_stdlib.cma.js";
"stdlib/base_internalhash_types.cma.js";
"stdlib/base.cma.js"

I checked the wrapping functions are correct.

@vouillon is there something else I should check, or is there something that can make the debugging easier? Thanks!

[vouillon]

@gpetiot It is not clear to me whether the error happens while loading a module or afterwards. Maybe you can have some debug output before and after a module is loaded. This is here, right? https://github.com/jonludlam/js_top_worker/blob/f83c32d71ee54b2d18b53d109933e44ed6054d78/lib/worker.cppo.ml#L218-L229 You can also add a Firebug.console##log e in the exception handler to see whether an exception is raised.

Are you able to load a module that does not contain any JavaScript code?

I would expect the error to happen on the worker side, but it seems to happen on the playground. But maybe something went wrong on the worker and the playground got a bogus message from the worker?

You should make sure your code is compiled with --pretty and not minified.

[hhugo]

@gpetiot, can you describe the build and loading process a bit.

• Is the toplevel built using whole program compilation or separate compilation
• How do you load new libraries ? from this PR, I understand that you prebuild cma.js files and simply load it in the browser after the toplevel is loaded/intialized. If so, I don't think this workflow is currently supported. I don't expect modules loaded this way to be visible in the toplevel.

[vouillon]

@hhugo I think the code for loading cma.js files is here: https://github.com/jonludlam/js_top_worker/blob/f83c32d71ee54b2d18b53d109933e44ed6054d78/lib/worker.cppo.ml#L185C1-L229 From what I heard, this is working when the files do not need additional runtime JavaScript code. @gpetiot is trying to make it work when some JavaScript code needs to be loaded as well.

[gpetiot]

@vouillon I added some debug output in js_top_worker (https://github.com/jonludlam/js_top_worker/compare/main...gpetiot:js_top_worker:debug-toplevel) but it doesn't show up in the browser debug console. I also checked that there is no minify and that we compile with --pretty but the errors are not more explicit.

@hhugo the toplevel is built at the same time as ocamlorg's playground (https://github.com/ocaml/ocaml.org/blob/main/playground/src/dune) from what I understand. The cma/cmi files are indeed linked during the toplevel initialization: https://github.com/gpetiot/ocaml.org/blob/pkg-toplevel/playground/src/main.ml#L48-L51 in the playground. That works fine when a package doesn't need a runtime.js (like astring).

Is there a way to support this workflow in the future, or is there a technical obstacle? This is an objective we have for both the tooling team and the ocaml.org team so we would really like to make it happen.

edit: I've summarized all the infos from issues/PRs/offline discussions in https://hackmd.io/@gpetiot/ByRuo1JlT to share with Tarides' tooling team

[hhugo]

@gpetiot, it would be much easier to validate this workflow with a simpler setup (E.g. Not using web workers)

[vouillon]

@gpetiot I'm trying to reproduce your issue, but I get this error when running voodoo-prep.

voodoo-prep: [INFO] runtime prep/universes/6/base/v0.16.1/voodoo_js_files/base/runtime.js: not found

It seems Jsoo_toplevel.copy_js_files does nothing. The following command (which is, as far as I can see, performed by Ocamlfind.js_files) returns nothing:

ocamlfind query -predicates byte,javascript -o-format -r base

I would have expected this command to get the list of runtime files:

ocamlfind query -format "%(jsoo_runtime)" base

And this one to get the directory in which these files can be found:

ocamlfind query -format "%d" base

[hhugo]

The javascript predicate was a hack that predates dune. Recent version of dune stopped emitting it very recently (dune.3.10 maybe).

To get the path of all js files for a set of findlib packages, you can use

ocamlfind query -format "%+(jsoo_runtime)" -r pkg1 pkg2 pkg3

[hhugo]

@vouillon, are you using an unreleased version of dune by any chance ? It is the unreleased dune.3.11 https://github.com/ocaml/opam-repository/pull/24510 that stop emitting the javascript predicate.

[hhugo]

@gpetiot, have you made any progress ?

[gpetiot]

No I didn't make any progress since my last comment. My team is deprioritizing this for the moment.

I will keep trying to fix this on the side, if someone finds a workaround I'm all ears :)

[hhugo]

I took a look at this. It seems to me that it would be simpler to also embed cmis files together with the runtime. It would make the logic found in https://github.com/jonludlam/js_top_worker/blob/f83c32d71ee54b2d18b53d109933e44ed6054d78/lib/worker.cppo.ml#L185C1-L229 unnecessary. @vouillon, what do you think ?

[hhugo]

I've pushed a few changes.

• --toplevel involves --linkall
• Fix symbol lookup (one should not need to set Clflags.no_check_prims := true)
• Make sure to emit embedded runtime only once per file

All this is still WIP and needs to be cleaned up but should unlock @gpetiot

[hhugo]

I've rebased this PR

• I've added an example in toplevel/example/lwt_toplevel. One can load the precompiled lib with #load_js "test_lib_jsoo.js". Calling Test_lib_jsoo.A.test () will use a js stubs included in the js file.
• I've cleaned up the implementation a bit.
• This still uses some es6 feature but I'd say let's revisit if someone complains.

@vouillon, would you have time to make a pass of review ?