worker.js file should be placed into the output directory when targeting Emscripten with pthread support

[syrel]

Problem

When targeting wasm32-unknown-emscripten with Pthread support, the *.worker.js file is not copied to the output directory from the deps folder, even though the Emscripten compiler generates that file. The worker.js is required by multithreaded wasm apps and is a mandatory output artefact.

Steps

# Create a test package
cargo new --bin issue && cd issue
# Compile with Pthreads support
cargo +nightly \
   -Z build-std=panic_abort,std \
   --config='build.rustflags = ["-C","target-feature=+atomics,+bulk-memory", "-C", "link-args=-pthread"]' \
   build \
      --bin issue \
      --target wasm32-unknown-emscripten
# List output files
find target/wasm32-unknown-emscripten/debug -maxdepth 1 -regex ".*issue.*"

Expected output:

target/wasm32-unknown-emscripten/debug/issue.d
target/wasm32-unknown-emscripten/debug/issue.js
target/wasm32-unknown-emscripten/debug/issue.wasm
target/wasm32-unknown-emscripten/debug/issue.worker.js

Actual output:

target/wasm32-unknown-emscripten/debug/issue.d
target/wasm32-unknown-emscripten/debug/issue.js
target/wasm32-unknown-emscripten/debug/issue.wasm

Note that issue.worker.js is missing in the output directory. However, it is successfully created by the compiler and can be found in the deps folder:

$ find target/wasm32-unknown-emscripten/debug/deps -maxdepth 1 -regex ".*issue.*"

target/wasm32-unknown-emscripten/debug/deps/issue.d
target/wasm32-unknown-emscripten/debug/deps/issue.js
target/wasm32-unknown-emscripten/debug/deps/issue.wasm
target/wasm32-unknown-emscripten/debug/deps/issue.worker.js

Possible Solution(s)

A solution would be to adjust the list of file types returned by file_types() for wasm32 target to include *.worker.js: https://github.com/rust-lang/cargo/blob/5da28587846d6cb2694e5bb1db1b5ca327285bf7/src/cargo/core/compiler/build_context/target_info.rs#L405

Notes

No response

Version

cargo 1.77.1 (e52e36006 2024-03-26)
release: 1.77.1
commit-hash: e52e360061cacbbeac79f7f1215a7a90b6f08442
commit-date: 2024-03-26
host: aarch64-apple-darwin
libgit2: 1.7.2 (sys:0.18.2 vendored)
libcurl: 8.4.0 (sys:0.4.70+curl-8.5.0 system ssl:(SecureTransport) LibreSSL/3.3.6)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Mac OS 14.3.1 [64-bit]

[weihanglo]

Thanks for the report!

Could you share some more specifications and documentations of that output behavior of emscripten? I found one here, but not sure if it always does that. worker.js could possibly be considered as an auxiliary file and uplifted for empscripten target, though we might need to understand the stability before moving forward. We might also bump into issues like https://github.com/rust-lang/cargo/issues/13355 and https://github.com/rust-lang/cargo/issues/13358 when dealing with it.

[weihanglo]

BTW, are there any more files for emscripten that we want to uplift to the target directory?

[syrel]

Hi @weihanglo

Thanks for a quick response! In the Special Considerations section it mentions that the worker.js will be always generated when compiling with -pthread flag and the file must be deployed:

Also note that when compiling code that uses pthreads, an additional JavaScript file NAME.worker.js is generated alongside the output .js file (where NAME is the basename of the main file being emitted). That file must be deployed with the rest of the generated code files.

In addition, -pthread is not the only case when Emscripten generates a worker.js. According to the emcc compiler manual , when linking with --proxy-to-worker flag it will also generate worker.js when emitting JavaScript. Cargo always emits JavaScript and never targets HTML when building for wasm32.

--proxy-to-worker [link] Runs the main application code in a worker, proxying events to it and output from it. If emitting HTML, this emits a .html file, and a separate .js file containing the JavaScript to be run in a worker. If emitting JavaScript, the target file name contains the part to be run on the main thread, while a second .js file with suffix “.worker.js” will contain the worker portion.

Emscripten choses what to emit based on a suffix of the output file: .js for JavaScript and .html for HTML. The following condition in cargo guarantees that Emscripten will emit JavaScript (due to .js suffix) and therefore we get a worker.js file. https://github.com/rust-lang/cargo/blob/5da28587846d6cb2694e5bb1db1b5ca327285bf7/src/cargo/core/compiler/build_context/target_info.rs#L405

Additional files

BTW, are there any more files for emscripten that we want to uplift to the target directory?

Yes indeed. Those files are: .data, .data.js and .mem. See Build Files. data related files are the important ones. They contain preloaded files and directories that will populate the virtual file system. See Packaging Files. It is difficult to envision an app without asset files. Please consider outputting them as well.

Cheers

[weihanglo]

Hey. Sorry for the delayed response.

The Cargo team discussed this during today's meeting. The general idea here is that we are okay for adding worker.js as an additional auxiliary files, since we already have special cases for something like .wasm.map.. For any other files at this moment, I'll encourage using --message-format=json to access artifact messages and process what you need if possible.

Just note that, due to the fact that wasm32-unknown-emscripten is a tier-2 target, as well as we lack of CI infra to test it, Cargo would provide its best-effort for the proposed behavior. There is no strong guarantee like tier-1.

In a better future, we'd like to see rustc informing Cargo about the files it emits, so that Cargo doesn't need to hard-code all of them. Thank you.

[charmitro]

A general note for the future, keep in mind of this PR that recently was merged in Emscripten.

With the master branch Emcscripten, *.worker.js file generates the following,

// This file is no longer used by emscripten and has been created as a placeholder                                                                                                           
// to allow build systems to transition away from depending on it.                                                                                                                           
//                                                                                                                                                                                           
// Future versions of emscripten will likely stop generating this file at all.

[weihanglo]

Thanks @charmitro for the updated info!

Given *.worker.js is no longer useful, I propose to close this issue.

[charmitro]

Thanks @charmitro for the updated info!

Given *.worker.js is no longer useful, I propose to close this issue.

I generally agree. However, is there a way to check which version is installed and, based on that, generate *.worker.js, at least until Emscripten completely removes this feature?

[weihanglo]

To me, I won't bother that. Emscripten is not a tier 1 platform and user can always download the latest version of it. The maintenance cost might not be worthy for a deprecated-soon-removed feature. And we don't have proper CI infra to test that. Also, I don't think there is a safe and stable way to get the version info. Invoking emscripnt is probably the straightforward one, and Cargo should invoke external binaries as less as possible.

[charmitro]

Makes sense, you're right. I completely agree with not moving forward with this.

[syrel]

Hi

What a timing of the emscripten change! Thanks @charmitro for digging it up. I think we an close the issue 👍

Cheers