SetReplace icon indicating copy to clipboard operation
SetReplace copied to clipboard

Only download Mathematica/Wolfram Engine once per workflow

Open maxitg opened this issue 4 years ago • 2 comments

The problem

Due to tests parallelization #578, we are now downloading the docker image with Mathematica 4 times, which is redundant. We should figure out how to only download it once thus saving 3 minutes' worth of credits. If we download it prior to starting the testing job, we will also reduce the workflow duration by 1 minute.

Possible solution

  1. Take WL out of the Docker image and put it into the Workspace for all nodes to use (tricky because it needs to stay activated).
  2. We can then add another wolfram-engine-download job which is going to put it into the workspace.
  3. The build/test jobs will then just need to put symbolic links/copy relevant files which should be much faster.

Alternative solutions

Circle CI has docker layer caching which might be useful here, however, it is not available on the free plan, and even if it was, it's quite expensive costing more credits than the entire WL testing job.

maxitg avatar Dec 17 '20 23:12 maxitg

Documentation for attach_workspace states that:

The full contents of the workspace are downloaded and copied into the directory the workspace is being attached at.

Therefore, if we persist the wolfram installation and attach on our four parallel jobs, it will download the wolfram installation on each one. DLC looks like it could be promising though.

aokellermann avatar Jan 13 '21 21:01 aokellermann

We should even be able to cache it between builds similar to how you cache Python modules.

maxitg avatar Jan 14 '21 16:01 maxitg