Reducing Open Files

[niederhuth]

I need to run a large analysis, however my university's cluster places a hard limit on the number of open files and is not exactly willing to alter this for me. Do you have any recommended work around...short of going to the cloud?

Thank you for your help. Specifics from the OrthoFinder error message are below:

ERROR: The system limits on the number of files a process can open is too low. For 454 species OrthoFinder needs to be able to open at least r=206216 files. Please increase the limit and restart OrthoFinder

1. Check the hard and soft limits on the number of open files for your system: $ ulimit -Hn $ ulimit -Sn
2. If hard limit, h > r already, then you just need to increase the soft limit: $ ulimit -n 206216
3. Alternatively, if h < r then you need to edit the file '/etc/security/limits.conf', this requires root privileges. To increase the limit to 206216 for user called 'emms' add the lines: emms hard nofile 206216 emms soft nofile 206216 (edit these lines to match your username)
4. Check the limit has now been updated (if you changed the hard limit you'll need to open a new session and confirm it's updated): $ ulimit -Sn
5. Once the limit is updated restart OrthoFinder with the original command

[chenyangkang]

Hi niederhuth,

I have a similar problem on my cluster. I am asking the cluster manager to modified the limits for me (up to 120000). But I'm not sure will it damage the system and I'm considering an alternative. I noticed that orthofinder can add a species each time, based on the previouse result. How about add one each time?

[niederhuth]

Hi chenyangkang,

I thought of that. However, 1) I have >1000 species to try and run! 2) when I tried adding species in batches, it looked to me as if it was still going to need to have all blast results open, so I killed the job before I ran into issues.

[chenyangkang]

Hi chenyangkang,

I thought of that. However, 1) I have >1000 species to try and run! 2) when I tried adding species in batches, it looked to me as if it was still going to need to have all blast results open, so I killed the job before I ran into issues.

Well, that's a giant program. Just for further information that you not only need to modifiy the file limits of the nodes, but also that of your PBS system (something call limits.pbs_mom), after which you need to restart the PBS system. Realy troublesome.

[davidemms]

Hi

There's not any functionality to reduce the number of open files required. Unfortunately, it would require quite a complex re-write of the ortholog inference code and would probably have performance implications too. At some point I might be able to find some time to see if I can provide an option but it wouldn't be any time soon.

Sorry your system admin isn't willing to increase the limit, it's usually not something they've done in the past and so can be apprehensive. Having done it on numerous machines now there have never been any issues. The reson orthofinder needs to open so many files is that there is an ortholog file for each species pair. Parallelisation is performed over the gene trees, and every gene tree can (and often does) have every species pair in it which means that for each gene tree analysed, orthologs need to be written to all of these files. I investigated opening and closing the files as required, but this puts more burden on the filesystem, and often checks by any anti-virus software can cause a delay in opening each file every time it's opened which can bring things crashing to a halt. This means it's more efficient in terms of resources to just open them once and write to them as required.

Best wishes David

[ghost]

Hi niederhuth,

I have a dataset of about 3,000 genomes and I am using Orthofinder for phylogenetic trees. However, the process takes longer than I expected. I wonder how large your dataset is and how long it takes for you to get the analysis done? I am using 72 cores on my univeristy's cluster.

[lgretton]

HPC admin here, not an OrthoFinder user. The very large number of file handles that are required for OF are really problematic on a large shared system. We have a user who's use of OF requires over 800k file handles and we just can't honour that. Consider that our single OF user has the ability to start thousands of such jobs, all hitting the same file system simultaneously. Keeping the open file limit count down on individual nodes allows us to minimise the performance impact this would have on the file system.

[davidemms]

Hi Liam

The latest release has an option --fewer-open-files that should resolve this: https://github.com/davidemms/OrthoFinder/releases/tag/2.5.5

Best wishes David

[davidemms]

(The option will become default in the 3.0 release)

[lgretton]

That looks excellent, I'm sorry I hadn't noticed it already.