nf-tower
nf-tower copied to clipboard
publishDir fails to publish files to s3 when >5gb
When the result of a process is larger than 5gb, the upload to s3 fails with the error:
The specified copy source is larger than the maximum allowable size for a copy source: 5368709120 (Service: Amazon S3; Status Code: 400; Error Code: InvalidRequest
Minimal reproducible example:
nextflow.enable.dsl=2
process createFile {
publishDir "s3://some-bucket", mode: 'copy', overwrite: true, failOnError: true
output:
path("*.gz")
shell:
'''
truncate -s 10G test.tar.gz
'''
}
workflow {
createFile()
createFile.out.view { "File: $it" }
}