Overcoming limitations on the number of reads that can be processed

[ishaanpbs]

Hi There seems to be a limitation in the number of reads that can be processed by porechop at a time, is there a way to remove this limit.

Thanks

[barneypotter24]

Hello, I would like to add that we are having a similar issue, resulting in Porechop failing on a library of ~5.5 million reads. Thanks

[rrwick]

I suspect this is a limitation of Porechop's somewhat simplistic design: it currently loads all reads into memory and then processes them. So if you have more reads than you can fit into memory, I'd expect it to crash. Does this fit with what you're seeing?

The workaround would be to split your input reads into multiple files and run Porechop on each - not very elegant but probably the only solution at the moment.

The real fix would be a redesign of how Porechop works. It could load reads as it handles them and then free up the memory when it's done with each read. This would obviously be better, but it would be a lot of work :smile: So I'll leave this issue open as an enhancement.