ggcat icon indicating copy to clipboard operation
ggcat copied to clipboard

GGCAT keeps panicking about too many open files

Open iosfwd opened this issue 1 year ago • 1 comments

I'm trying to run GGCAT on the following dataset: https://ftp.ebi.ac.uk/pub/databases/ENA2018-bacteria-661k/ I had 5T of free disk space and the following flags: -k 31 -m 32 -j 8 -s 1 However, my runs keep panicking on too many open files despite raising ulimit -n. My first two runs with ulimit -n of 1024 and 4096 crashed in the following phase:

Started phase: kmers merge prev stats:
Thread panicked at location: libs-crates/parallel-processor-rs/src/memory_fs/file/internal.rs:248:26
Error message: called `Result::unwrap()` on an `Err` value: Os { code: 24, kind: Uncategorized, message: "Too many open files" }
Backtrace:    0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: <unknown>
  16: <unknown>
  17: <unknown>
  18: <unknown>
  19: <unknown>
  20: <unknown>
  21: <unknown>
  22: <unknown>

Then I raised the ulimit -n to 10240, and it crashed in the following phase:

Finished phase: kmers merge. phase duration: 20675.80s gtime: 30013.67s
Started phase: hashes sorting prev stats: 
Finished phase: hashes sorting. phase duration: 646.23s gtime: 30659.90s
Started phase: links compaction prev stats: 
Thread panicked at location: libs-crates/parallel-processor-rs/src/memory_fs/file/internal.rs:248:26
Error message: called `Result::unwrap()` on an `Err` value: Os { code: 24, kind: Uncategorized, message: "Too many open files" }
Backtrace:    0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: <unknown>
  16: <unknown>
  17: <unknown>
  18: <unknown>
  19: <unknown>
  20: <unknown>
  21: <unknown>
  22: <unknown>

iosfwd avatar May 24 '23 07:05 iosfwd