Hardcoded cache key prevents multiple workflows or jobs from caching emsdk

[lopopolo]

The cache key for the GitHub Actions cache of emsdk is hardcoded:

https://github.com/mymindstorm/setup-emsdk/blob/ca0cef33e53bf9149891246a1b13ef3dbd498ca5/src/main.ts#L28

This prevents multiple CI jobs or workflows from using this action and caching the results.

Can this key include the workflow name and job? Alternatively, can overriding it be exposed as an action parameter?

The code samples in the README seem to indicate that users should use actions/cache on their own. Why does this action also invoke @actions/cache?

[mymindstorm]

This prevents multiple CI jobs or workflows from using this action and caching the results.

I fail to see how the current cache key prevents the use of a cache across multiple jobs. If a job finishes and has the same emsdk version and architecture, it should be fetched successfully by other jobs.

Can this key include the workflow name and job? Alternatively, can overriding it be exposed as an action parameter?

You can manually use actions/cache and set actions-cache-folder. If actions-cache-folder is already present and contains emsdk, the action will not create or fetch a cache.

The code samples in the README seem to indicate that users should use actions/cache on their own. Why does this action also invoke @actions/cache?

If I remember correctly, system libraries may be generated after emsdk is used to build files. The action is done running at this point, so if you want to cache these files, you will need to manually use actions/cache.

[lopopolo]

I got this error when using this action in multiple concurrent workflows (separate YAML files)

[mymindstorm]

I am interpreting 'error' as 'it did not use the cache during a concurrent workflow'.

I would assume that this would be the case for any job started before at least one job creates the cache. If this is showing across multiple workflows with the same cache key, could you share some example runs and your configuration?

[lopopolo]

This is on my main branch after I merged a PR to use this action in two separate workflows. The job was aborted with an error.

Build Playground
Unable to reserve cache with key 2.0.9-upstream-x64-master, another job may be creating this cache.
Build Playground
No cached files found at path "/home/runner/work/playground/playground/emsdk-cache" - downloading and caching emsdk.

Log

https://github.com/artichoke/playground/runs/3257374864?check_suite_focus=true

Warning: No cached files found at path "/home/runner/work/playground/playground/emsdk-cache" - downloading and caching emsdk.
/usr/bin/unzip -q /home/runner/work/_temp/7da39686-e10e-4e0d-9eab-a7e75f2824c4
/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/emsdk update-tags
`update-tags` is not longer needed.  To install the latest tot release just run `install tot`
/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/emsdk install 2.0.9-upstream
Resolving SDK version '2.0.9' to 'sdk-releases-upstream-d8e430f9a9b6e87502f826c39e7684852f59624f-64bit'
Installing SDK 'sdk-releases-upstream-d8e430f9a9b6e87502f826c39e7684852f59624f-64bit'..
Installing tool 'node-14.15.5-64bit'..
Downloading: /home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/zips/node-v14.15.5-linux-x64.tar.xz from https://storage.googleapis.com/webassembly/emscripten-releases-builds/deps/node-v14.15.5-linux-x64.tar.xz, 21391232 Bytes
 [----------------------------------------------------------------------------]
Unpacking '/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/zips/node-v14.15.5-linux-x64.tar.xz' to '/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/node/14.15.5_64bit'
Done installing tool 'node-14.15.5-64bit'.
Installing tool 'releases-upstream-d8e430f9a9b6e87502f826c39e7684852f59624f-64bit'..
Downloading: /home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/zips/d8e430f9a9b6e87502f826c39e7684852f59624f-wasm-binaries.tbz2 from https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/d8e430f9a9b6e87502f826c39e7684852f59624f/wasm-binaries.tbz2, 280846232 Bytes
 [----------------------------------------------------------------------------]
Unpacking '/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/zips/d8e430f9a9b6e87502f826c39e7684852f59624f-wasm-binaries.tbz2' to '/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/upstream'
Done installing tool 'releases-upstream-d8e430f9a9b6e87502f826c39e7684852f59624f-64bit'.
Done installing SDK 'sdk-releases-upstream-d8e430f9a9b6e87502f826c39e7684852f59624f-64bit'.
/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/emsdk activate 2.0.9-upstream
Resolving SDK version '2.0.9' to 'sdk-releases-upstream-d8e430f9a9b6e87502f826c39e7684852f59624f-64bit'
Setting the following tools as active:
   node-14.15.5-64bit
   releases-upstream-d8e430f9a9b6e87502f826c39e7684852f59624f-64bit

Next steps:
- To conveniently access emsdk tools from the command line,
  consider adding the following directories to your PATH:
    /home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main
    /home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/node/14.15.5_64bit/bin
    /home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/upstream/emscripten
- This can be done for the current shell by running:
    source "/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/emsdk_env.sh"
- Configure emsdk in your shell startup scripts by running:
    echo 'source "/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/emsdk_env.sh"' >> $HOME/.bash_profile
/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/emsdk construct_env
Adding directories to PATH:
PATH += /home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main
PATH += /home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/upstream/emscripten
PATH += /home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/node/14.15.5_64bit/bin

Setting environment variables:
PATH = /home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main:/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/upstream/emscripten:/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/node/14.15.5_64bit/bin:/opt/hostedtoolcache/Ruby/3.0.2/x64/bin:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/home/runner/.local/bin:/opt/pipx_bin:/usr/share/rust/.cargo/bin:/home/runner/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/runner/.dotnet/tools:/snap/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
EMSDK = /home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main
EM_CONFIG = /home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/.emscripten
EMSDK_NODE = /home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/node/14.15.5_64bit/bin/node
export PATH="/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main:/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/upstream/emscripten:/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/node/14.15.5_64bit/bin:/opt/hostedtoolcache/Ruby/3.0.2/x64/bin:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/home/runner/.local/bin:/opt/pipx_bin:/usr/share/rust/.cargo/bin:/home/runner/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/runner/.dotnet/tools:/snap/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games";
export EMSDK="/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main";
export EM_CONFIG="/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/.emscripten";
export EMSDK_NODE="/home/runner/work/_temp/a4456bf8-0169-45e6-a19c-af0aaabe6816/emsdk-main/node/14.15.5_64bit/bin/node";
Error: Unable to reserve cache with key 2.0.9-upstream-x64-master, another job may be creating this cache.

[philass]

Is there any update on this? This is also an issue I am experiencing.

[hoodmane]

I am also running into this issue.

[hoodmane]

It's not hard to work around though: make a separate setup-emsdk-cache job which just runs setup-emsdk and then specify that every other job needs the setup-emsdk-cache job.

[ahlinc]

Also hit to this issue: https://github.com/tree-sitter/tree-sitter/actions/runs/4624890814/jobs/8180161810#step:5:67 It would be better to give cache key construction to outer scope. @mymindstorm please don't do this: https://github.com/mymindstorm/setup-emsdk/blob/ab889da2abbcbb280f91ec4c215d3bb4f3a8f775/src/main.ts#L31 A code like the next is totally enough to create the cache folder in a workspace folder:

const fullCachePath = emArgs.actionsCacheFolder; 

Also it allows to specify a full path to a custom location and prevents situations like bellow:

Warning: No cached files found at path "/home/runner/work/tree-sitter/tree-sitter/home/runner/work/_temp/emsdk-cache" - downloading and caching emsdk.

[mymindstorm]

@ahlinc I'll try to take a look at this on Saturday.

[dsvensson]

@mymindstorm What did you find during that Saturday? :)

[mymindstorm]

Oops, sorry.

[dsvensson]

@mymindstorm Maybe Sunday is a better day? :)

Just giving the option to control cache key to the user ought to be a very quick hack if you're short on time?

[akien-mga]

Confirming the issue on godotengine/godot, we've been running into this on and off when merging two PRs shortly after each other, and both would start their Web CI build around the same time. This leads to annoying failed builds. They can thankfully be restarted from the GH Actions UI so it's not too bad.

But now we added a second concurrent Web build to our CI pipeline, so we're always going to start two builds in parallel, which is going to lead to seeing this issue many times per day. So we'll really need a workaround. I'll look into what we can implement downstream.

Here's an example failure: https://github.com/akien-mga/godot/actions/runs/7567395177/job/20606547300

Error: Unable to reserve cache with key 3.1.39-linux-x64-master, another job may be creating this cache.

[limbonaut]

@akien-mga Here's a workaround I use to build Godot: https://github.com/limbonaut/limboai/blob/caa1508feaeba34f7fecb6cf17eac85873a4c84b/.github/workflows/web.yml#L35

[akien-mga]

@akien-mga Here's a workaround I use to build Godot: https://github.com/limbonaut/limboai/blob/caa1508feaeba34f7fecb6cf17eac85873a4c84b/.github/workflows/web.yml#L35

Thanks! That would work to mitigate the issue with a build matrix having multiple Emscripten jobs indeed. I suppose it wouldn't solve the issue when merging two separate PRs shortly after each other, where the two separate PR workflows might both run the init_emscripten job at the same time.

[mymindstorm]

Apologies for the wait. I have released v14, which should address the issues raised here.