nipype

nipype copied to clipboard

Summary

Support for FUSE filesystem for workdir

Actual behavior

Workdir reads and writes many intermediate files, with higher than normal latency. This leads to workflows failing due to files not present. (FUSE file systems keep a copy of the file in RAM and writes once the handle is closed. But the actual time it takes the files to be present for a new process can vary).

Expected behavior

Nipype, should have a delay mechanism on file read fails possibly with exponential backoff.

Platform details:

I tested this in GCP, with gcsfuse. There were random failures of workflow files not being present.

Execution environment

GCP machine, with gcsfuse mounted workdir. Singularity container installed with required tools with nipype.

@sulantha2006 - there is an execution config option (job_finished_timeout) that you can increase checking results files and other things.

https://miykael.github.io/nipype_tutorial/notebooks/basic_execution_configuration.html#Execution

Sure I will try, but is this param works when SGE/PBS or any other execution plugin is not used? I run the pipeline as workflow.run().

any distributed plugin if i remember correctly, but may have been created specifically for slurm/pbs/etc. however, i would not expect an issue with fuse necessarily on a single system (since the local fuse mapping should always be ok), only across systems.

but perhaps you are noticing this on a single system, and not using a batch system like slurm. are you running into this error when running the container in a given machine, with simply pointing at a working directory that is over fuse. one option would be to make the work dir local and then copy it over if necessary.

So my setup is a slurm cluster but I don't install anything on the controller or login node. I use it only to submit jobs.

I submit a slurm job per image I have to process which calls a python script inside a container. This script is a nipype workflow. Container has all tools and nipype installed.

The workdir is a mounted fuse dir bound to the container. I also have a data input and datasink mounted the same way. Didnt see any issues with those.

Each compute node is dynamic with a small local storage so local workdir is not possible. Also I want to keep the workflow files without modifying to skip reprocessing time consuming tasks. Also, if I were to transfer all workdir content before a job and after a job, I assume it would significantly increase cloud costs.

The failures are random. Out of 950 jobs, about 30-50 failed with some intermediate file in the workdir missing (Output of a previous node). Others completed successfully.

My assumption is that, the previous node output is read before the file is written and present through the fuse system.

Ideal solution should have a delayed retry for node inputs ? Is it easy to implement?

On Thu, Jan 7, 2021, 11:11 AM Satrajit Ghosh [email protected] wrote:

any distributed plugin if i remember correctly, but may have been created specifically for slurm/pbs/etc. however, i would not expect an issue with fuse necessarily on a single system (since the local fuse mapping should always be ok), only across systems.

but perhaps you are noticing this on a single system, and not using a batch system like slurm. are you running into this error when running the container in a given machine, with simply pointing at a working directory that is over fuse. one option would be to make the work dir local and then copy it over if necessary.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nipy/nipype/issues/3288#issuecomment-756320403, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHZL55ZNU27V37HJ36IUCTSYYBM5ANCNFSM4VZLPXDQ .

@sulantha2006 - thank you for the details. in this case it is then interesting that the failure happens - i would not have expected this given how things work.

also the job_finished_timeout will not help you here. the local job is likely using multiproc plugin, which does not have such a wait period. this would require a retry-timeout check in the _get_inputs function.

https://github.com/nipy/nipype/blob/master/nipype/pipeline/engine/nodes.py#L552

every node retrieves its inputs to evaluate the hash before computing or moving on. so this would be the place to check.

I am not sure the above PR solution would work. It will solve the issue of input files not being ready surely, but I am getting other fails now on the same root cause. See the above exception. Note that as before this is completely random.

/process_workdir/ is the gcsfuse mounted folder.

Traceback: 
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/nipype/pipeline/plugins/linear.py", line 46, in run
    node.run(updatehash=updatehash)
  File "/usr/local/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 512, in run
    write_node_report(self, is_mapnode=isinstance(self, MapNode))
  File "/usr/local/lib/python3.9/site-packages/nipype/pipeline/engine/utils.py", line 117, in write_node_report
    report_file.parent.mkdir(exist_ok=True, parents=True)
  File "/usr/local/lib/python3.9/pathlib.py", line 1312, in mkdir
    self._accessor.mkdir(self, mode)
OSError: [Errno 5] Input/output error: '/process_workdir/GCP/MK_BL_Proc/902-234_Baseline/temporary/main_wf/t1_wf/t1_extract/_report'