Store Nextflow head process generated files

[MrMarkW]

Description

Nextflow has options to generate timeline reports and dags on the completion of a workflow. They are default written to the directory the engine is running. I couldn't find a simple way to have those file written to the context/project s3 bucket.

Use Case

This useful so we can ingest the consolidated trace, timeline, report and dags from the workflow provide by nextflow.

Proposed Solution

Allow configuration to copy files in default or specific relative directory on head node to the s3 work dir bucket.

[heuermh]

This works for us in nf-core on AWS Batch and also with Tower; I haven't tried with Amazon Genomics CLI yet.

Perhaps having this block in your config might help?

timeline {
  overwrite = true
}
report {
  overwrite = true
}
trace {
  overwrite = true
}
dag {
  overwrite = true
}

See e.g. https://github.com/nf-core/configs/blob/master/conf/awsbatch.config#L12

[MrMarkW]

I will try this again. I also might not have been using the publishDir correctly.

[elswob]

Did you get this to work?

[github-actions[bot]]

Greetings! Sorry to say but this is a very old issue that is probably not getting as much attention as it deserves. We encourage you to check if this is still an issue in the latest release and if you find that this is still a problem, please feel free to open a new one.

[mmueller76]

Hi @MrMarkW

This can be achieved by either by including the following in your nextflow.config:

    report {
      enabled = true
      file = 's3://<agc-bucket>/<output-prefix>/report.html'
    }

    trace {
      enabled = true
      file = 's3://<agc-bucket>/<output-prefix>/trace.txt'
    }

    timeline {
      enabled = true
      file = 's3://<agc-bucket>/<output-prefix>/timeline.html'
    }

    dag {
      enabled = true
      file = 's3://<agc-bucket>/<output-prefix>/dag.html'
    }

or this in your MANIFEST.json

{
  "mainWorkflowURL": "main.nf",
  "inputFileURLs": [
    "inputs.json"
  ],
  "engineOptions": "-with-report s3://<agc-bucket>/<output-prefix>/report.html -with-trace s3://<agc-bucket>/<output-prefix>/trace.txt - with-timeline s3://<agc-bucket>/<output-prefix>/timeline.html -with-dag s3://<agc-bucket>/<output-prefix>/dag.html"
}