UX improvements to explore running processes

[GeigerJ2]

One usability complaint that is brought up frequently is that AiiDA makes it rather difficult to explore running workflows (on the HPC). This is, for example, because of the non human-readable run directories on the HPC that don't follow an organizational pattern that resembles the workflow logic.

For example, to explore every calcjob of a workflow on the HPC, one would go through the following cycle:

verdi process list/show/status  # get what you need, typically the <pk>
verdi calcjob gotocomputer  # retrieve info from HPC, or go directly onto HPC
<exit ssh session>  # on the HPC, impossible to know the directory of the next calcjob in the workflow
verdi process list/show/status  # get the next <pk>
verdi calcjob gotocomputer
... etc. (repeat for every CalcJob in your workflow)

Hence, this issue aims to compile possible ways to make this easier. I propose the endpoint verdi process explore.

We solved the issue outlined above in another project, by having a create-symlink-tree CLI endpoint that, on the HPC, creates symlinks to the (non human-readable) run directories of all calcjobs in a workflow, in a structure that resembles the workflow. This looks something like this:

[santis][jgeiger@santis-ln002 workflows]$ pwd
/capstor/scratch/cscs/jgeiger/aiida/workflows
[santis][jgeiger@santis-ln002 workflows]$ ls
WorkGraph-AQUAPLANET-5642
[santis][jgeiger@santis-ln002 workflows]$ ll WorkGraph-AQUAPLANET-5642/
total 24
drwxr-x---+ 2 jgeiger mr32 4096 Nov  5 08:37 ./
drwxr-x---+ 3 jgeiger mr32 4096 Nov  5 08:37 ../
lrwxrwxrwx  1 jgeiger mr32   74 Nov  5 08:37 cleanup -> /capstor/scratch/cscs/jgeiger/aiida/7d/97/aa29-f192-4b8d-8d2b-0ef8a236e536/
lrwxrwxrwx  1 jgeiger mr32   74 Nov  5 08:37 icon_date_2000_01_01_00_00_00 -> /capstor/scratch/cscs/jgeiger/aiida/51/5b/253a-592d-4800-a9bb-8675b09679e4/
lrwxrwxrwx  1 jgeiger mr32   74 Nov  5 08:37 icon_date_2000_01_01_00_00_10 -> /capstor/scratch/cscs/jgeiger/aiida/8e/50/25c7-be39-424c-8f30-e2a6a285de7d/
lrwxrwxrwx  1 jgeiger mr32   74 Nov  5 08:37 icon_date_2000_01_01_00_00_20 -> /capstor/scratch/cscs/jgeiger/aiida/bd/27/9e58-14c4-44fb-9f0b-9335542cc9c8/

In addition, every directory gets a symlink to the workflow root folder and, possibly, the previous and next (if finished) calculations in the workflow. We'd like to bring this feature into aiida-core. Please add additional ideas below, so we can convert them into sub-issues.

• [ ] CLI endpoint to create a symlink tree for the whole workflow on the HPC with human-readable directory names and organization that follows the workflow logic
• [ ] Possibly mount the created workflow directory with symlinks on the local machine (@alberto-carta; does the mounting work with symlinks on the HPC?)
• [ ] Allow verdi calcjob outputcat to display results of still running calculations (@npaulish)