Output uploading

[kislyuk]

Sorry if I'm missing something obvious.

I'd like to be able to specify a destination for outputs to be sent to (ideally even as soon as they appear among the task outputs):

task foo {
  output {
    File bar = "path/to/bar.bam"
  }
}

miniwdl run foo.wdl --dest-dir s3://my-bucket/results-for-foo/

should result in bar.bam appearing at s3://my-bucket/results-for-foo/path/to/bar.bam.

This should presumably use the same interface as the downloader plugins.

[kislyuk]

Alternative naming suggestion for the option: --remote-output-dir/--remote-out

[mlin]

This is a stopgap solution which should unblock the requirement until there's something more built-in: https://gist.github.com/mlin/55bafb76c784164651687336cd15e162

[mlin]

Further to the previous example upload_output_files.sh which shows how to upload all the output files on workflow completion,

The new example plugin s3_progressive_upload de-risks how to upload intermediate results as tasks complete, and generate a final workflow output JSON referring to the uploaded S3 objects. Of course integration work will still be needed for production use.