Error using http-multipart-body-parser to upload images

[vjmartinez89]

Describe the bug When I pass my request through the middleware and try to write the buffer as a file in my server-side code, its file size is considerably higher than the original one, and the file itself is corrupted. Plain text files are processed and stored without any problem.

To Reproduce How to reproduce the behaviour:

1. Sample code

import httpMultipartBodyParser from '@middy/http-multipart-body-parser'
...
export const handler = middy(lambdaHandler)
.use(httpHeaderNormalizer())
.use(httpMultipartBodyParser())
.use(httpErrorHandler());

Code to write the file:

import { createWriteStream } from 'fs';
... // get file content from body
createWriteStream(`./uploads/${filename}`).write(content);

Expected behaviour The file content I receive on my server is the same as I send in my request. The file resulting from writing the content buffer is not corrupt.

Environment (please complete the following information):

• Node.js: 16.19.1
• Middy: @middy/http-multipart-body-parser 4.6.4

[willfarrell]

Thanks for reporting! Do you think you could write a failing unit test? Take an attempt at a PR to resolve?

[willfarrell]

I did a little digging, might be related to the Content-Type header. We use busboy to parse the request body.

https://github.com/mscdex/busboy/issues/341

[willfarrell]

Closing due to lack of feedback. Please reopen if adding Content-Type doesn't resolve the issue.

[paulmaszlik]

I experience the same issue. I send a PNG file (26,403 bytes) via Postman (multpart/form-data, query_image is the file) to my local serverless endpoint. Node: 20.11.0 / WebPack: 5.89.0 / Serverless: 3.38.0

If I use the following middy packages: (I also tried with older v4 packages) @middy/[email protected] @middy/[email protected] @middy/[email protected]

Both plugins called before lambdaHandler via .use().

const lambdaHandler = async (event) => {
  console.log(event);
   ...
}

Console result:

{
  body: {
    query_image: {
      filename: 'testImage.png',
      mimetype: 'image/png',
      encoding: '7bit',
      truncated: false,
      content: <Buffer c2 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 00 71 00 00 00 c2 9c 08 06 00 00 00 48 c2 a4 44 c2 8c 00 00 66 c3 aa 49 44 41 54 78 c2 9c c3 ... 39695 more bytes>
    }
  },
  cookies: [],
  headers: {
    'user-agent': 'PostmanRuntime/7.36.0',
    accept: '*/*',
    'postman-token': 'cf2cfa05-c351-4ef5-adbf-f06849b5a2e3',
    host: 'localhost:3000',
    'accept-encoding': 'gzip, deflate, br',
    connection: 'keep-alive',
    'content-type': 'multipart/form-data; boundary=--------------------------466455162022947297222544',
    'content-length': '26620'
  },
  ...

---

If I use a similar multipart/form-data processor package [email protected] (npm) I get the correct file size and attachment is processable:

const parser = require('lambda-multipart-parser');

const lambdaHandler = async (event) => {
  const result = await parser.parse(event);
  console.log(result.files);
   ...
}

Result:

[
  {
    content: <Buffer 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 00 71 00 00 00 9c 08 06 00 00 00 48 a4 44 8c 00 00 66 ea 49 44 41 54 78 9c ed fd 77 bc 64 47 79 ... 26353 more bytes>,
    filename: 'testImage.png',
    contentType: 'image/png',
    encoding: '7bit',
    fieldname: 'query_image'
  }
]

---

Both packages use busboy, slight differences. Not sure what causes the problem.

[willfarrell]

Looking at the two implementations:

lambda-multipart-parser

 file.on('data', data => {
            uploadFile.content = data;
        });

This means it can only handle files up to a certain size.

@middy/http-multipart-body-parser

file.on('data', (data) => {
          chunks.push(data)
        })
        file.on('end', () => {
          attachment.content = Buffer.concat(chunks)
        })

To support larger files, Buffer.concat concat is used here.

Note: c2 added at the beginning and every so often for some reason

@paulmaszlik can you create a PR with a unit test?

[ajaz-rehman]

just pass charset: 'binary' to the middleware like this:

httpMultipartBodyParser({
  charset: "binary",
})

[willfarrell]

@ajaz-ur-rehman Thanks, I'll add this to the docs.