[GCP] Visualization of Nextflow pipeline execution

[NShaforostov]

Background

Current Run logs page allows to view different run information like timings, instance info, run parameters, console output, etc. But in some cases, current view could be insufficient - for example, for Nextflow runs. For such runs, it could be useful to view separate Nextflow tasks and processes, their metrics and logs.

We shall integrate the visualization of Nextflow pipeline execution to the Cloud Pipeline GUI with the Nextflow execution engine to provide reporting of:

• Process statuses
• Task statuses
• CPU, memory, and disk utilization per task
• Command, parameters, and logs per task

Approach

Visualization of the Nextflow pipeline execution shall be being performed at the Run logs page, in the separate tab or panel. Possible view of such panel:

Panel shall include sections:

• Processes - section that displays the full list of Nextflow run processes and summary statuses of tasks in them
• Task statuses - section that displays list of Nextflow task statuses of all/selected processes as an horizontal bar graph
• Tasks - section that displays list of tasks and their metrics of all/selected processes as a table

By default, sections shall display all available Nextflow processes and tasks.

User shall have the ability:

• to click a specific process in the Processes section:
  • to view summary over statuses of the tasks of that process (in the Task statuses section)
  • to view list of tasks of that process (in the Tasks section)
• to click a specific task in the Tasks section, to view task details in the separate pop-up

Task details pop-up shall contain tabs:

• CLI - the command that is executed in the selected task
• Parameters - all non-empty task metrics like CPU and memory usage, realtime of the task execution, disk usage, etc.
• Execution log - task execution logs