pydra icon indicating copy to clipboard operation
pydra copied to clipboard

Version cache dirs independently from Pydra version

Open effigies opened this issue 7 months ago • 1 comments

via https://github.com/nipype/pydra/issues/149#issuecomment-2892241979

Since the run-cache now gets created /Users/<user>/Library/Caches/pydra/1.0a1/run-cache by default (by platformdirs on Mac, will be different on Windows and Linux), (need separate issue about versioning caches), we could specify a directory called /Users/<user>/Library/Caches/pydra/1.0a1/runs (or similar), which contains timestamped symlinks to the top-level cache directory that was submitted, e.g. /Users/<user>/Library/Caches/pydra/1.0a1/runs/20250520065430, which the last run could be picked up when a specific cache directory isn't specified by pydra crash

Just a quick note that I think we should version the caches in their own right, not based on the Pydra version.

You can see this style of thing in effect in uv, where each type of thing that might be cached has its own version format. Some need to be frequently bumped, others haven't:

$ ls .cache/uv 
archive-v0  CACHEDIR.TAG     git-v0          sdists-v9   simple-v16
builds-v0   environments-v2  interpreter-v4  simple-v15  wheels-v5

We could do something like .cache/pydra/tasks-v0 for now, and then when a change to the engine justifies a cache invalidation, we bump to tasks-v1.

effigies avatar May 19 '25 20:05 effigies