chexpert-labeler icon indicating copy to clipboard operation
chexpert-labeler copied to clipboard

Published 20 hours ago • verified publisher icon stanfordmlgroup

Out of memory

Open catfish132 opened this issue 2 years ago • 2 comments

my csv is so big that the memory is out of the range. May be there is a way to fix out

catfish132 avatar Sep 27 '22 06:09 catfish132

my csv is so big that the memory is out of the range. May be there is a way to fix out

Maybe try reading the CSV in chunks, running the labeler on each chunk, then writing the output iteratively.

jirvin16 avatar Sep 27 '22 16:09 jirvin16

I faced the same issue as I was processing 1.5M reports. I created a set of scripts to chunk a large file and process chunks in parallel automatically. This might be useful for others who see this so they don't have to reinvent the wheel:

https://github.com/BardiaKh/ParallelCheXpertLabeler

BardiaKh avatar Jul 29 '23 23:07 BardiaKh