aiida-core icon indicating copy to clipboard operation
aiida-core copied to clipboard

Add `--also-remote` option to `verdi process dump`

Open GeigerJ2 opened this issue 1 year ago • 2 comments

Tries to open a connection to the remote Computer using the transport of the CalcJobNode, and gets the files and folders which were not originally dumped, either as they weren't part of the CalcJobNode's repository or retrieved.

Will add the tests once the implementation is somewhat stable.

Don't want to assign you officially for review, but still pinging @khsrali and @agoscinski here if you guys want to have a look 👀

Note: With PR #6584, it'll be possible to give a warning to the user about the total size of files that would be written to disk. We might eventually decide that it takes too long and one has to open too many transports, just to obtain the total file size to warn a user, so one could just issue a generic warning. The problem is that these files and directories are on the remote computer (if not deleted already), so one has to connect to get information about them.

Also, the returned file size will differ, as some of the files in the directory to which the RemoteData points to, will have already been retrieved. This can be checked for, based on e.g. retrieve_list, local_copy_list, etc.

GeigerJ2 avatar Oct 03 '24 11:10 GeigerJ2

Codecov Report

Attention: Patch coverage is 27.58621% with 21 lines in your changes missing coverage. Please review.

Project coverage is 77.82%. Comparing base (ef60b66) to head (5693a63). Report is 115 commits behind head on main.

Files with missing lines Patch % Lines
src/aiida/tools/dumping/processes.py 24.00% 19 Missing :warning:
src/aiida/cmdline/commands/cmd_process.py 50.00% 2 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6578      +/-   ##
==========================================
+ Coverage   77.51%   77.82%   +0.31%     
==========================================
  Files         560      566       +6     
  Lines       41444    42077     +633     
==========================================
+ Hits        32120    32741     +621     
- Misses       9324     9336      +12     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Oct 03 '24 11:10 codecov[bot]

Thanks @GeigerJ2 I like this feature, We already had some users from Empa who probably would have liked this..

khsrali avatar Oct 03 '24 12:10 khsrali