OSError: [Errno 28] No space left on device

[diplockn]

When running Circle-Map Realign in a conda environment Circle-Map Realign -t 24 -i sorted_candidates.A.bam -qbam qcircle.A.bam -sbam sorted_circle.A.bam -fasta ref.fa -o final_circle.A.bed

I get an OSError: File "/global/home/users/diplockn/.conda/envs/circle/lib/python3.7/site-packages/pybedtools/bedtool.py", line 3256, in cat TMP.write(str(f)) OSError: [Errno 28] No space left on device

This doesn't appear to be a ram issue since the cluster I'm running on does not give an out of memory error

I get the same error when using python /global/home/users/diplockn/Circle-Map/circlemap/circle_map.py Realign -t 24 ...

[iprada]

Dear @diplockn,

Thanks for posting your question. As you pointed out, the issue is not RAM problem. The problem is caused because apparently Circle-Map is writing a substantial amount of temporary files on disk. In the end, if you exceed the amount of available temp disk space, you get the error. Coincidentally, the issue#27 found a similar problem a week ago.

To solve your issue: At the stage the software is right now:

I suggest you to run Circle-Map using less threads. I think that setting it to 16 should reduce the amount of temp files on disk, avoiding the OSError.

Long term solution:

You and another user (issue #27 ) have reported similar problems. I have tried to reproduce the error on my cluster, but it seems to work fine for me. However, if this causes problems to many users it should be handled and fixed at some point. At this moment I am busy with other projects (there is a PhD that needs to be finished). But if you or anybody else reading this keeps getting the error after reducing the number of threads, write a comment and I will increase the priority to fix the problem.

Best wishes,

Iñigo

[diplockn]

Hey Iñigo,

It seems like Circle-Map Realign puts temporary files in the home directory of the cluster I'm working on by default. I used TMPDIR to make a temporary directory in our clusters scratch storage:

source activate circlemap export TMPDIR=/global/scratch/diplockn/${sample} mkdir -p $TMPDIR Circle-Map Realign -t 24 -i sorted_candidates.${sample}.bam -qbam qcircle.${sample}.bam -sbam sorted_circle.${sample}.bam -fasta /global/scratch/users/diplockn/ref.fa -o final_circle.${sample}.bed conda deactivate

and it worked no problem with 24 threads.

Best,

Nate

[iprada]

Dear Nate,

Sounds great. Glad to hear that changing the tempdir worked. I did not consider changing the TMP folder to a custom folder during the whole software execution. But to be honest, it seems a very reasonable way of handling the issue. Thanks a lot for the idea. I am going to leave the issue open until I fix the problem.

Best,

Iñigo

[johnstonmj]

Hi Iñigo,

If you have yet to adjust the directory for temporary files, I just thought that I'd mention that the TMPDIR solution above only redirects some of the temporary files.

I am modifying TMPDIR as diplockn does above. Within my TMPDIR location, I see many "pybedtools.xxxxxxxx.tmp" files, as well as a couple "pymp-xxxxxxxx" directories. However, in my working directory, I also see multiple directories of "temp_files_xxxxx".

Would it be possible to generate all temporary files in the same location? This would be beneficial on my cluster setup, where I have a large storage allowance specifically for temporary files. Additionally, it would stray file clean-up easier in case a compute job terminates early.

Thanks as always, Michael