OrthoFinder
OrthoFinder copied to clipboard
Reduce number of required file handles
I have created a small hack to reduce the number of file handles.
It uses python stringio to to everything in memory and just write the files on close.
To enable it use USE_MEM=1 as environment var
It will use much more memory but sometimes this is easier to archive as to change ulimits
@davidemms @Phhere Why open all files at the same time? Is it much slower to open the necessary files in append mode? Something like this:
def WriteOlogLinesToFile(file: str, text: str, lock: Lock):
if len(text) == 0:
return
if debug:
util.PrintTime("Waiting: %d" % os.getpid())
lock.acquire()
try:
if debug:
util.PrintTime("Acquired lock: %d" % os.getpid())
with open(file, 'a') as f:
f.write(text)
finally:
lock.release()
if debug:
util.PrintTime("Released lock: %d" % os.getpid())