lune icon indicating copy to clipboard operation
lune copied to clipboard

io.lines() equivalent for better iteration over files

Open mxruben opened this issue 1 year ago • 3 comments

Hello, I am currently looking to write some tools with lune that deal with very large files. The only option for opening files is to read the entire thing into a string in memory. This works fine for smaller files, but when the file is gigabytes large (10 GB in my case), it becomes a real issue. If the fs library could open a file pointer and had something like vanilla lua's io.lines for iteration, it would solve the cases where reading an entire file into a string isn't feasible.

mxruben avatar Aug 06 '24 22:08 mxruben

I wrote an implementation for this here. I can make a PR if the design for this is improved or we can discuss.

I have a use case that makes this useful. @filiptibell opinions?

CleverSource avatar Dec 23 '24 04:12 CleverSource

Hey! As a part of #211, I implemented a basic async stream system for the new process.create API. Can we abstract the stream implementation into the lune-utils crate and extend it for this sort of chunked reads of files?

That way, we can expand more built-ins with streaming functionality relatively easily.

CompeyDev avatar Dec 23 '24 08:12 CompeyDev

Sounds good to me. I'll update my changes when possible and create a PR when it's ready.

CleverSource avatar Dec 23 '24 09:12 CleverSource