Caching and remote folders

[greschd]

Here's a problem I encounter somewhat regularly: Calculations which pass (part of) their outputs on as a RemoteData input are broken once their scratch directory is cleaned. This is a normal tradeoff - if we don't want to spend the long-term cost of keeping the data around (in the repository), we have to accept sometimes having to re-run a calculation.

It becomes problematic however when caching is introduced: When trying to re-run the calculation (to regenerate the remote folder), caching will happily use the existing calculation (without a remote folder). As a result, subsequent calculations which use the remote folder will fail.

The solution to this problem I can think of is to use the is_valid_cache mechanism to disable caching when the remote folder is gone. This should probably not be the default, but something that can be decided by the calculation class (which should know if the remote folder is used in this way).

An additional layer of complexity comes into play when we consider a chain of calculations that are all cached. Now, we again want to use the cache, because the calculation that would have needed the remote folder doesn't actually need to run. Here's a complicated way this could be solved: When a RemoteData output of a cached calculation is used as an input to a calculation (which is actually trying to run), this triggers a check: If the data is gone, the cached calculation needs to first be run. That would propagate up the chain of cached calculations as needed. I'm not sure if this is reasonably possible (or even desirable) though.

[broeder-j]

I would agree that the plugin can/should handle this, or where ever the logic for the usage of the remote folder is. Maybe it can even get the file/information somehow else, so it might not always be the case that such a calculation is always broken for every plugin.

I think the 'chain of calculations' within a workchain might also be handleable (marking calc in retrospect as is_valid_cache or not. We are currently discussing to 'clean up' the repository to some extent for intermediate calculations in workchains (to save disk space in HTC runs), which would disallow to use these calculations as further starting points (same thing as if remote folder not there).

For other cases certainly an automatic solution on the aiida side might be nice, through I am also not sure if this is desirable (because one cannot know in the extreme case how many calculations will be launched with the submission of just one), one should therefore be able to switch it off. Your proposed solution could also apply for non cashed calculations. If I want to rerun calc 2, which has (cleaned) remote folder of (not cashed) calc 1 as input.

[ltalirz]

Just mentioning that in AiiDA 2, Node.is_valid_cache is now a setter as well https://github.com/aiidateam/aiida-core/pull/5207