modules icon indicating copy to clipboard operation
modules copied to clipboard

Published 20 hours ago • verified publisher icon nf-core

XeniumRanger import-segmentation failed on Tower

Open an-altosian opened this issue 6 months ago • 2 comments

Have you checked the docs?

Description of the bug

Dear @khersameesh24 and @maxulysse,

I am trying to run the proseg subworkflow from spatialxe on Nextflow Tower, which uses the nf-core xeniumranger module to convert proseg results to a xenium bundle. I have tested that this error was persistent.

    File "/xeniumranger-xenium3.1/mro/insitu/stages/xr_select_dataset/__init__.py", line 47, in main
      os.link(str(src_path), outs.dataset)
  OSError: [Errno 95] Operation not supported: '/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET/XENIUM_RANGER_CS/XR_IMPORT_TRANSCRIPT_ASSIGNMENTS/fork0/join-u008637f7fd/files/cells.zarr.zip' -> '/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET/XENIUM_RANGER_CS/SELECT_CELLS_DATASET/fork0/chnk0-u008637f958/files/cells.zarr.zip'

Command used and terminal output

$ nextflow run main.nf -profile docker --input samplesheet.csv --outdir results --segmentation proseg --coordinate_based

The exit status of the task that caused the workflow execution to fail was: 1

Error executing process > 'PROSEG:XENIUMRANGER_IMPORT_SEGMENTATION (XET)

Caused by:
  Essential container in task exited


Command executed:

  xeniumranger import-segmentation \
      --id="XET" \
      --xenium-bundle="output-XET" \
       \
       \
       \
       \
      --transcript-assignment="xr-transcript-metadata.csv" \
      --viz-polygons="xr-cell-polygons.geojson" \
      --units=microns \
      --localcores=94 \
      --localmem=760 \
  
  
  cat <<-END_VERSIONS > versions.yml
  "PROSEG:XENIUMRANGER_IMPORT_SEGMENTATION":
      xeniumranger: $(xeniumranger -V | sed -e "s/xeniumranger-/- /g")
  END_VERSIONS

Command exit status:
  1

Command output:
  2025-05-29 04:45:03 [runtime] (chunks_complete) ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_SUBSAMPLE_TRANSCRIPTS
  2025-05-29 04:45:03 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_SUBSAMPLE_TRANSCRIPTS.fork0.join
  2025-05-29 04:51:04 [runtime] (update)          ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_SUBSAMPLE_TRANSCRIPTS.fork0 join_running
  2025-05-29 04:57:05 [runtime] (update)          ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_SUBSAMPLE_TRANSCRIPTS.fork0 join_running
  2025-05-29 05:03:06 [runtime] (update)          ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_SUBSAMPLE_TRANSCRIPTS.fork0 join_running
  2025-05-29 05:04:21 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_SUBSAMPLE_TRANSCRIPTS
  2025-05-29 05:40:12 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_FINALIZE_RELABEL
  2025-05-29 05:40:13 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_FINALIZE_RELABEL.fork0.chnk0.main
  2025-05-29 05:40:46 [runtime] (chunks_complete) ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_FINALIZE_RELABEL
  2025-05-29 06:00:29 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS
  2025-05-29 06:00:29 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS.fork0.split
  2025-05-29 06:00:29 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.XR_UNZIP_TRANSCRIPTS
  2025-05-29 06:00:29 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.XR_UNZIP_TRANSCRIPTS.fork0.chnk0.main
  2025-05-29 06:00:32 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS
  2025-05-29 06:00:33 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS.fork0.join
  2025-05-29 06:06:15 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS
  2025-05-29 06:06:16 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.MERGE_METRICS
  2025-05-29 06:06:16 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.MERGE_METRICS.fork0.chnk0.main
  2025-05-29 06:06:16 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.SELECT_CELLS_DATASET
  2025-05-29 06:06:16 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.SELECT_CELLS_DATASET.fork0.chnk0.main
  2025-05-29 06:06:19 [runtime] (chunks_complete) ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.MERGE_METRICS
  2025-05-29 06:06:20 [runtime] (failed)          ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.SELECT_CELLS_DATASET
  
  [error] Pipestance failed. Error log at:
  XETG00509__0043420__Kidney_1__20241219__000427/XENIUM_RANGER_CS/SELECT_CELLS_DATASET/fork0/chnk0-u008637f958/_errors
  
  Log message:
  Traceback (most recent call last):
    File "/xeniumranger-xenium3.1/external/martian/adapters/python/martian_shell.py", line 648, in _main
      stage.main()
    File "/xeniumranger-xenium3.1/external/martian/adapters/python/martian_shell.py", line 607, in main
      self._run(lambda: self._module.main(args, outs))
    File "/xeniumranger-xenium3.1/external/martian/adapters/python/martian_shell.py", line 564, in _run
      cmd()
    File "/xeniumranger-xenium3.1/external/martian/adapters/python/martian_shell.py", line 607, in <lambda>
      self._run(lambda: self._module.main(args, outs))
    File "/xeniumranger-xenium3.1/mro/insitu/stages/xr_select_dataset/__init__.py", line 47, in main
      os.link(str(src_path), outs.dataset)
  OSError: [Errno 95] Operation not supported: '/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET/XENIUM_RANGER_CS/XR_IMPORT_TRANSCRIPT_ASSIGNMENTS/fork0/join-u008637f7fd/files/cells.zarr.zip' -> '/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET/XENIUM_RANGER_CS/SELECT_CELLS_DATASET/fork0/chnk0-u008637f958/files/cells.zarr.zip'
  
  
  Waiting 6 seconds for UI to do final refresh.
  Pipestance failed. Use --noexit option to keep UI running after failure.
  
  2025-05-29 06:06:26 Shutting down.
  Saving pipestance info to "/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET/XETG00509__0043420__Kidney_1__20241219__000427.mri.tgz"
  For assistance, upload this file to 10x Genomics by running:
  
  xeniumranger upload <your_email> "/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET/XET.mri.tgz"

Command error:
  (more omitted..)
  2025-05-29 05:03:06 [runtime] (update)          ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_SUBSAMPLE_TRANSCRIPTS.fork0 join_running
  2025-05-29 05:04:21 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_SUBSAMPLE_TRANSCRIPTS
  2025-05-29 05:40:12 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_FINALIZE_RELABEL
  2025-05-29 05:40:13 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_FINALIZE_RELABEL.fork0.chnk0.main
  2025-05-29 05:40:46 [runtime] (chunks_complete) ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_FINALIZE_RELABEL
  2025-05-29 06:00:29 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS
  2025-05-29 06:00:29 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS.fork0.split
  2025-05-29 06:00:29 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.XR_UNZIP_TRANSCRIPTS
  2025-05-29 06:00:29 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.XR_UNZIP_TRANSCRIPTS.fork0.chnk0.main
  Could not read cgroup stats: open /sys/fs/cgroup/memory/ecs/12cd279ff2f54605ab1483f664a883cb/46b378d1a297945e96ffed9d65d8bd76180aa4b3cb5d565f0381bc6aa6bcc577/memory.stat: no such file or directory
  tar: Removing leading `/' from member names
  2025-05-29 06:00:32 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS
  2025-05-29 06:00:33 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS.fork0.join
  2025-05-29 06:06:15 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS
  2025-05-29 06:06:16 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.MERGE_METRICS
  2025-05-29 06:06:16 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.MERGE_METRICS.fork0.chnk0.main
  2025-05-29 06:06:16 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.SELECT_CELLS_DATASET
  2025-05-29 06:06:16 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.SELECT_CELLS_DATASET.fork0.chnk0.main
  2025-05-29 06:06:19 [runtime] (chunks_complete) ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.MERGE_METRICS
  2025-05-29 06:06:20 [runtime] (failed)          ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.SELECT_CELLS_DATASET
  [error] Pipestance failed. Error log at:
  XETG00509__0043420__Kidney_1__20241219__000427/XENIUM_RANGER_CS/SELECT_CELLS_DATASET/fork0/chnk0-u008637f958/_errors
  Log message:
  Traceback (most recent call last):
    File "/xeniumranger-xenium3.1/external/martian/adapters/python/martian_shell.py", line 648, in _main
      stage.main()
    File "/xeniumranger-xenium3.1/external/martian/adapters/python/martian_shell.py", line 607, in main
      self._run(lambda: self._module.main(args, outs))
    File "/xeniumranger-xenium3.1/external/martian/adapters/python/martian_shell.py", line 564, in _run
      cmd()
    File "/xeniumranger-xenium3.1/external/martian/adapters/python/martian_shell.py", line 607, in <lambda>
      self._run(lambda: self._module.main(args, outs))
    File "/xeniumranger-xenium3.1/mro/insitu/stages/xr_select_dataset/__init__.py", line 47, in main
      os.link(str(src_path), outs.dataset)
  OSError: [Errno 95] Operation not supported: '/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET/fork0/join-u008637f7fd/files/cells.zarr.zip' -> '/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET/SELECT_CELLS_DATASET/fork0/chnk0-u008637f958/files/cells.zarr.zip'
  Waiting 6 seconds for UI to do final refresh.
  Pipestance failed. Use --noexit option to keep UI running after failure.
  2025-05-29 06:06:26 Shutting down.
  Saving pipestance info to "/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET.mri.tgz"
  For assistance, upload this file to 10x Genomics by running:
  xeniumranger upload <your_email> "/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XETG.mri.tgz"
  Fusion Info:
      ami-id: ami-0161203d1ed74d5f9
      instance-id: i-05713dd0e069ef5b6
      instance-type: r6id.32xlarge
      fusion_version: 2.4.13-cd14732
      clone_namespace: false
      kernel_version: 4.14
      disk_cache_size: 49Gb
      max_open_files: 1048576

Work dir:
  s3://scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336

Container:
  wave.seqera.io/wt/87676aa08cd1/nf-core/xeniumranger:3.1.1

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

Relevant files

I could run the workflow locally, but it failed on Tower with fusion turned on. The error was a Python "Operation not supported" error when XeniumRange was trying to use os.link to create a hard link. The complete error message is attached below. As I remember we encountered similar problem in nf-core scrnaseq with cellranger, here I also tag the amazing nf-core scrnaseq developers in case they have any idea. Thanks. @grst @fmalmeida @apeltzer.

System information

No response

an-altosian avatar Jun 02 '25 16:06 an-altosian

@fntlnz , this looks like a fusion issue, can you suggest a fix?

SPPearce avatar Jun 29 '25 11:06 SPPearce

@an-altosian @SPPearce unfortunately hard links are not supported by the fusion filesystem and the xeniumranger tool uses them inside. xeniumranger is not open source so apparently the only possibility is to do something on the fusion side. I'll do a pass with the team on this.

fntlnz avatar Jul 08 '25 09:07 fntlnz

Even though not as efficient, would it work to set the nextflow scratch directive to have Xeniumranger work outside the fusion FS?

grst avatar Jul 09 '25 15:07 grst

Is it possible to check whether fusion is enabled within the module, and to exit gracefully in that case?

SPPearce avatar Jul 09 '25 15:07 SPPearce

Yes I could specify scratch=true to make it work, but still, this is definitely not the best option. We should either address this long standing problem in fusion, or ask 10x Genomics to avoid using hard link. I bet they did the similar thing in CellRanger because cell ranger works fine with fusion.

an-altosian avatar Jul 09 '25 18:07 an-altosian

Yes I could specify scratch=true to make it work, but still, this is definitely not the best option. We should either address this long standing problem in fusion, or ask 10x Genomics to avoid using hard link. I bet they did the similar thing in CellRanger because cell ranger works fine with fusion.

I feel asking 10x Genomics to avoid using hard link is an easy thing to do, they may not do it but it is easy to ask.

SPPearce avatar Jul 09 '25 18:07 SPPearce

Yes passing process.scratch = true is the only way to make it work for now. We are planning on doing a refactor to make hardlinks possible but it's not something that will happen in the short term.

fntlnz avatar Jul 14 '25 11:07 fntlnz

@khersameesh24 Based on the discussion above, maybe the easiest thing to do is to set scratch in the module based on the executor, like what quantms does?

scratch if (executor == "local") { false } else { true }

https://github.com/bigbio/quantms/blob/b17b26b3a0ddf83c6bdaa97c72d60dd867a49ba2/modules/local/thermorawfileparser/main.nf#L13

an-altosian avatar Jul 29 '25 17:07 an-altosian