SetReplace
SetReplace copied to clipboard
Only download Mathematica/Wolfram Engine once per workflow
The problem
Due to tests parallelization #578, we are now downloading the docker image with Mathematica 4 times, which is redundant. We should figure out how to only download it once thus saving 3 minutes' worth of credits. If we download it prior to starting the testing job, we will also reduce the workflow duration by 1 minute.
Possible solution
- Take WL out of the Docker image and put it into the Workspace for all nodes to use (tricky because it needs to stay activated).
- We can then add another
wolfram-engine-download
job which is going to put it into the workspace. - The build/test jobs will then just need to put symbolic links/copy relevant files which should be much faster.
Alternative solutions
Circle CI has docker layer caching which might be useful here, however, it is not available on the free plan, and even if it was, it's quite expensive costing more credits than the entire WL testing job.
Documentation for attach_workspace
states that:
The full contents of the workspace are downloaded and copied into the directory the workspace is being attached at.
Therefore, if we persist the wolfram installation and attach on our four parallel jobs, it will download the wolfram installation on each one. DLC looks like it could be promising though.
We should even be able to cache it between builds similar to how you cache Python modules.