joystick icon indicating copy to clipboard operation
joystick copied to clipboard

Upgrade multer dependency so we can leverage streams

Open rglover opened this issue 2 years ago • 2 comments

Noticed that uploads can be slow and after digging found out that the latest version of Multer switched to using streams for uploads. Just need to refactor the uploads API and make sure we keep the shape of the returned data consistent (e.g., req.file or req.files still exists).

Another option is to just use Busboy directly (the core dependency Multer uses to parse uploads). Technically we don't need a bunch of fancy features—we just want the file data.

rglover avatar Oct 01 '23 17:10 rglover

Just did a test. Only "gotcha" is making sure we handle the write to disk from the stream for local uploads and then pass the stream off to S3 for uploads there: https://stackoverflow.com/questions/37336050/pipe-a-stream-to-s3-upload.

rglover avatar Oct 01 '23 17:10 rglover

Got it. See examples in your /cheatcode/experiments/multer2 directory.

rglover avatar Oct 01 '23 18:10 rglover

I ended up trying this but the latest releases completely change data structures and the streaming doesn't work how you'd think. It basically just writes the file to a temp folder on disk and then creates a stream from that. Not worth it compared to the current version.

rglover avatar Jul 24 '24 18:07 rglover