Running sratools prefetch on Azure

[drpatelh]

Description of the bug

The exit status of the task that caused the workflow execution to fail was: 1

Error executing process > 'NFCORE_FETCHNGS:SRA:SRA_FASTQ_SRATOOLS:SRATOOLS_PREFETCH (SRR14709033)'

Caused by:
  Process `NFCORE_FETCHNGS:SRA:SRA_FASTQ_SRATOOLS:SRATOOLS_PREFETCH (SRR14709033)` terminated with an error exit status (1)

Command executed:

  eval "$(vdb-config -o n NCBI_SETTINGS | sed 's/[" ]//g')"
  if [[ ! -f "${NCBI_SETTINGS}" ]]; then
      mkdir -p "$(dirname "${NCBI_SETTINGS}")"
      printf '/LIBS/GUID = "5d0db162-e6a9-4167-a826-10b11c40e7e1"\n/libs/cloud/report_instance_identity = "true"\n' > "${NCBI_SETTINGS}"
  fi
  
  retry_with_backoff.sh prefetch \
       \
      --progress \
      SRR14709033
  
  vdb-validate SRR14709033
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_FETCHNGS:SRA:SRA_FASTQ_SRATOOLS:SRATOOLS_PREFETCH":
      sratools: $(prefetch --version 2>&1 | grep -Eo '[0-9.]+')
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  mkdir: can't create directory '/home/_azbatchtask_1/': Permission denied

Work dir:
  az://pipelines/work/b2/8953f0f0c34a6644be0f7ee8a84b21

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

Command used and terminal output

Pipeline configuration:

{
    "description" : "Pipeline to fetch metadata and raw FastQ files from public databases.",
    "launch" : {
        "pipeline" : "https://github.com/nf-core/fetchngs",
        "revision" : "1.5",
        "configProfiles" : [ "test" ],
        "paramsText" : "outdir: 'az://pipelines/profile_test/fetchngs/results'"
    }
}
``

Relevant files

Configuration used to create Azure environment:

{
  "workDir" : "az://pipelines/work",
  "region" : "eastus2",
  "preRunScript" : "export NXF_VER=21.10.4",
  "forge" : {
    "vmType" : "Standard_E8ds_v4",
    "vmCount" : 60,
    "autoScale" : true,
    "disposeOnDeletion" : true
  },
  "deleteJobsOnCompletion" : "on_success",
  "deletePoolsOnCompletion" : false,
  "environment" : [ ],
  "discriminator" : "azure-batch"
}

System information

Nextflow version 21.10.4 Hardware: Azure Container engine: Docker Version of nf-core/fetchngs: 1.5

[Midnighter]

Maybe users on Azure can define NCBI_SETTINGS to a writable path? Something like

env.NCBI_SETTINGS = '/common/writable/path/vdb.conf'

I don't have access to facilities to test this.

[adamrtalbot]

I don't see this on dev anymore. On 1.9 I get a new, interesting error. Links on Tower: 1.9 dev

[adamrtalbot]

Given this now works on dev I'd say this issue will be fixed with the next release (v1.10)