replace-in-file icon indicating copy to clipboard operation
replace-in-file copied to clipboard

Add support for streaming large files

Open mykola-mokhnach opened this issue 5 years ago • 4 comments

Hello Adam,

Thanks for the great library, it's really useful. I'd like to mention one thing about its usage: when trying to perform replacements in big files (like hundreds of megabytes) then OutOfMemory errors could easily pop up, because the lib loads the file content fully into the memory before, actually, making the replacement. I'm not quite sure whether this was done by design, but it, probably, makes sense to at least mention this fact in README.

An alternative solution, which is memory-friendly (uses streams), could be something like https://github.com/eugeneware/replacestream

mykola-mokhnach avatar Sep 16 '20 10:09 mykola-mokhnach

Hi there, yep some work on this was started in https://github.com/adamreisnz/replace-in-file/issues/40, but that has eventually been abandoned.

I'd be happy to review and accept PR's that implement some kind of streaming option or support for large files.

However, I'm not sure how the replace algorithm will work correctly if the file contents are read in chunks (for example, the regex may not find a match when a chunk is split exactly at a match).

adamreisnz avatar Sep 18 '20 23:09 adamreisnz

Hello @adamreisnz I have a solution implemented and tested, but i don't have permissions to create a branch.

Noeldesarrollo avatar Jun 15 '21 12:06 Noeldesarrollo

Hi @Noeldesarrollo , please fork the repository and then open a pull request, and I'll be able to have a look. Thanks!

adamreisnz avatar Jun 15 '21 22:06 adamreisnz

PR: https://github.com/adamreisnz/replace-in-file/pull/139

Noeldesarrollo avatar Jun 16 '21 11:06 Noeldesarrollo