pgsc_calc icon indicating copy to clipboard operation
pgsc_calc copied to clipboard

Nextflow pipeline fails due to missing .pgen file, despite file being present.

Open Fiwx opened this issue 1 year ago • 1 comments

Description of the bug

Pipeline execution:

  • Initial run: Failed, error reported missing .pgen file.
  • Rerun with '-resume': Same error occurred.
  1. Versions: Nextflow 24.04.4, Java 11.0.24, PLINK2 v2.00a3.
  2. Disk space: 212G free out of 280G.
  3. Work directory contents: .pgen file present (85711355 bytes).
  4. File permissions: -rw-r--r-- for .pgen file.
  5. Nextflow log: Confirms error about missing .pgen file.
  6. PLINK2 log: Shows successful execution, reports writing .pgen file.
  7. Environment: Singularity 4.1.1 used for containerization.

I ran the command to replicate the issue in the work directory and it ran this:

plink2 \
    --threads 2 \
    --memory 16384 \
    --set-all-var-ids '@:#:$r:$a' \
    --max-alleles 2 \
    --freq \
    --missing vcols=fmissdosage,fmiss \
    --new-id-max-allele-len 100 missing \
    --vcf use1n.vcf.gz  \
    --allow-extra-chr --chr 1-22, X, Y, XY \
    --make-pgen vzs pvar-cols="-xheader,-maybequal,-maybefilter,-maybeinfo,-maybecm" \
    --out GRCh37_use1n_ALL

Result: Command executed successfully, .pgen file created and visible in directory. I had to select "y" asking it to unzip some files (since other files were already present).

  • .pgen file exists but Nextflow unable to locate it.
  • No apparent file system, permission, or disk space issues.
  • PLINK2 reports successful file creation in both pipeline and manual execution.
  • Manual execution creates file without issues.

Similar to https://github.com/PGScatalog/pgsc_calc/issues/282 but this most recent issue is happening on the newest release.

Fiwx avatar Sep 12 '24 18:09 Fiwx

Which Nextflow process fails? Could you please copy the output from your terminal and upload .command.log / .command.err / .command.run from the working directory of the failed process?

Does the conda profile work around the issue? If it does, then it's a singularity configuration problem (which nextflow should automatically handle, but there will be edge cases).

nebfield avatar Sep 13 '24 08:09 nebfield