w3cli icon indicating copy to clipboard operation
w3cli copied to clipboard

Don't load CAR file into memory during upload

Open makew0rld opened this issue 1 year ago • 5 comments

Command: w3 up --car my_file.car.

When running this command with a large file, for example a CAR file created from a 4 GiB file of random data, I noticed my memory usage would go up by ~4 GiB, indicating w3 is loading the entire file into memory during the upload process.

I'm not sure exactly where in the code this is coming from, but it's not ideal and could cause OOM crashes/issues in some scenarios. Please let me know if a fix for this is possible.

makew0rld avatar Jun 10 '24 18:06 makew0rld