Should the `local_copy_list` files take precedence?

[mbercx]

Currently the files from the remote_copy_list are copied to the remote after "putting" the files from the local_copy_list:

https://github.com/aiidateam/aiida-core/blob/a2a05a69fb2fa6aae9a96d49d543e72008d2888f/aiida/engine/daemon/execmanager.py#L245-L268

This was causing issues for me while working around a bug in the epw.x code where I have to set outdir to ./. Basically the aiida.in file of the previous calculation would overwrite the one generated by the prepare_for_submission script.

Obviously the issue here is with the epw.x code, but it got me thinking that perhaps we should put the files prepared in the local sandbox folder from the local_copy_list after copying those of the remote_copy_list. I would think that in most use cases the user would want the local_copy_list files to take precedence. A first implementation for restarts would typically just copy the entire folder contents of a parent calculation, and currently this would fail in case the input file names are the same (which would almost always be true for BaseRestartWorkChain restarts).

[sphuber]

I also encountered this problem in aiida-shell. I think it might make sense to have the order be:

1. CalcInfo.remote_copy_list
2. CalcInfo.local_copy_list
3. Contents of the SandboxFolder

I think this makes more sense intuitively and it gives more control to the user because the remote_copy_list interface is quite limited and so you cannot control finely how files are copied (for example through exclude filters or renaming files), but with the sandbox, you can do whatever you want.

The only problem, clearly, is backwards compatibility. Is it possible that some plugins out there actively relied on this behavior?

[mbercx]

The only problem, clearly, is backwards compatibility. Is it possible that some plugins out there actively relied on this behavior?

It's hard to say. I'd expect few developers to rely consciously on this behaviour, but I suppose it could break things. Maybe something to add to

https://github.com/aiidateam/aiida-core/issues/6157

For me at least it's not an urgent issue to fix.