Node.js axios upload large file using FormData, read all data to the memory, cause out of memory issue.

[MengLi619]

Describe the bug

Node.js axios upload large file using FormData, read all data to the memory, cause out of memory issue.

To Reproduce

  const formData = new FormData();
  formData.append('file', fs.createReadStream(path), { filepath: path, filename: basename(path) });
  formData.append('videoId', videoId);
  await axios.post(UPLOAD_MEDIA_URL, formData, {
    headers: {
      'Content-Type': `multipart/form-data; boundary=${formData.getBoundary()}`,
      'Authorization': `Bearer ${await token()}`,
    },
    maxBodyLength: Infinity,
    maxContentLength: Infinity,
  });

Expected behavior

Read file and upload by stream, without loading all data in the memory.

Environment

• Axios Version [0.21.1]
• Adapter [HTTP]
• Node.js Version [v14.16.0]
• OS: [Alpine in docker, OSX 10.15.6]
• Additional Library Versions [[email protected]]

Additional context/Screenshots

Not needed.

[MengLi619]

Just give a clue for it, after I add the maxRedirects: 0 option, the upload will use much less memory than before. maxRedirects: 0 will switch the transport from http/https to follow-redirects, it seems follow-redirects use much more memory when uploading big file. Hope anyone can optimize this problem, thanks.

[jimmywarting]

Can't stretch how important it could be to have a spec compatible FormData in place. form-data dose a few things wrong and unexpectedly. and the formdata isn't even reusable in a friendly manner. that is why we have started to deprecate the use of form-data in node-fetch

If axios had one proper spec compatible formdata then it wouldn't be necessary to have to accumulate all data into memory as it would be possible to calculate the final content-length length and also re-read the formdata over and over again. with eg formdata-polyfill

if you where to create a Blob out of a FormData containing one very large file using my formdata-polyfill package. then it would just hold some few bytes in memory cuz nothing is read

// sample:
import { fileFromSync } from 'fetch-blob/from.js'
import { FormData, formDataToBlob } from 'formdata-polyfill/esm.min.js'

// creates a File like object (same as browser)
// and it dose not hold any data in memory, it's just a references
// point to where it can read the data to/from
const file = fileFromSync('./movie.mkv') 
const file2 = fileFromSync('./dump.bin') 

const fd = new FormData()
fd.append('movie', new Blob([file, file2]))
const blob = formDataToBlob(fd)

// still, nothing out of the gb large file have been read into memory
// and yet you can have a blob with a very huge size and and a `blob.stream()` method

This is thanks to how blob are built up internally and keeping references points to where it should be reading the data from

[mufteev]

if you where to create a Blob out of a FormData containing one very large file using my formdata-polyfill package. then it would > just hold some few bytes in memory cuz nothing is read

Hello, I have a similar problem, how can I solve this problem by applying the above code?

Node.js:

import { Blob } from 'node:buffer';
import { fileFromSync } from 'fetch-blob/from';
import { FormData, formDataToBlob } from 'formdata-polyfill/esm.min';
// ...
  const rs = fileFromSync(tmpPath); // Large file
  const fd = new FormData();
  // @ts-ignore
  fd.append(name, new Blob([rs]), filename);
  const blob = formDataToBlob(fd);
  // ...
  axios.post('http://localhost:3030/file', blob.stream(), {
    maxBodyLength   : Infinity,
    maxContentLength: Infinity,
    // headers: {
    //   'Content-Type': `multipart/form-data; boundary=${form.getBoundary()}`
    // }
  });

I'm trying to send data in this way, but it doesn't work =(

[jimmywarting]

I don't think axios can quite work with whatwg streams, blobs or 100% spec compatible formdata implementations on the backend. witch is a bit sad. I think you have to kind of do:

import { Blob } from 'node:buffer'
import { Readable } from 'node:stream'
import { fileFromSync } from 'fetch-blob/from.js'
import { FormData, formDataToBlob } from 'formdata-polyfill/esm.min.js'

const file = fileFromSync(tmpPath) // Large file
const fd = new FormData()

fd.append(name, file, filename)
const blob = formDataToBlob(fd)
const nodeStream = Readable.from(blob.stream())

axios.post('http://localhost:3030/file', nodeStream, {
  maxBodyLength   : Infinity,
  maxContentLength: Infinity,
  headers: {
    'content-type': blob.type,
    'content-length': blob.size
  }
})

(not tested)

[mufteev]

Run your example

blob.type //  - Ok
// << multipart/form-data; boundary=----6108568729206242634834187191
blob.size  // - Ok
// << 621848812

But for some reason the request itself is not executed, I track the receipt of the request from the next application on port :3030, but nothing happens

My code:

import axios from 'axios';
import { Blob } from 'node:buffer';
import { Readable } from 'node:stream';
import { fileFromSync } from 'fetch-blob/from';
import { FormData, formDataToBlob } from 'formdata-polyfill/esm.min';
// ...
            const rs = fileFromSync(tmpPath);
            const fd = new FormData();
            fd.append(name, rs, filename);
            const blob = formDataToBlob(fd);
            const nodeStream = Readable.from(blob.stream());

            console.log(blob.type);
            console.log(blob.size);

            axios.post('http://localhost:3030/file', nodeStream, {
              maxBodyLength   : Infinity,
              maxContentLength: Infinity,
              headers         : {
                'content-type'  : blob.type,
                'content-length': blob.size
              }
            });
// ...

[mufteev]

The situation is similar if I use http.request

import { request } from 'http';
import { Blob } from 'node:buffer';
import { Readable } from 'node:stream';
import { fileFromSync } from 'fetch-blob/from';
import { FormData, formDataToBlob } from 'formdata-polyfill/esm.min';
// ...
            const rs = fileFromSync(tmpPath);
            const fd = new FormData();
            fd.append(name, rs, filename);
            const blob = formDataToBlob(fd);
            const nodeStream = Readable.from(blob.stream());

            const _req = request('http://localhost:3030/file', {
              method : 'POST',
              headers: {
                'Content-Type'  : blob.type,
                'Content-Length': blob.size
              }
            });

            nodeStream.pipe(_req);
            _req.end(); // ?

If I don't call _req.end() the request is not executed, but if the call is made, the application on the port accepts an already closed stream as well as Axios:

{
  accept: 'application/json, text/plain, */*',
  'content-type': 'multipart/form-data; boundary=----5748967022509554367644258628',
  'content-length': '621848812',
  'user-agent': 'axios/0.26.1',
  host: 'localhost:3030',
  connection: 'close'
}

If you make a request to the application :3030 via Postman, the header will be like this:

{
  'user-agent': 'PostmanRuntime/7.29.0',
  accept: '*/*',
  'cache-control': 'no-cache',
  'postman-token': '4d52f5a2-6ce7-47a8-b1a2-1288d02b960b',
  host: 'localhost:3030',
  'accept-encoding': 'gzip, deflate, br',
  connection: 'keep-alive',
  'content-type': 'multipart/form-data; boundary=--------------------------030737611013686369151959',
  'content-length': '4621'
}

[pbell23]

@jimmywarting Thank you for your help. I also tried your code to reduce memory usage when uploading large files but I can't make it work. Could you post a working example to upload using blob streams ?

Code tested :

import * as path from 'node:path';

import axios from 'axios';
import { fileFromSync } from 'fetch-blob/from.js';
import { FormData, formDataToBlob } from 'formdata-polyfill/esm.min.js';

const file = fileFromSync(tmpPath); // Large file
const filename = path.basename(tmpPath);

const fd = new FormData();
fd.append('file', file, filename);
const blob = formDataToBlob(fd);

const response = axios.post(url, blob.stream(), {
    maxBodyLength: Infinity,
    maxContentLength: Infinity,
    headers: {
        'Content-Type': blob.type,
        'Content-Length': blob.size
    }
});

Results in an error :

node:internal/process/promises:279
            triggerUncaughtException(err, true /* fromPromise */);
            ^
AxiosError: socket hang up
    at connResetException (node:internal/errors:692:14)
    at TLSSocket.socketOnEnd (node:_http_client:478:23)
    at TLSSocket.emit (node:events:539:35)
    at endReadableNT (node:internal/streams/readable:1345:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  code: 'ECONNRESET',

[jimmywarting]

Not able to test right now, cuz i don't have a computer at hand now. But i believe that axioms do not support whatwg web streams. So you have to convert it to a node stream using streamReadable.from(blob.stream())

[jimmywarting]

Btw, node18 have FormData built in. It does also support 3th party blob impl like fetch-blob if you do wish to use built in fetch api

[d11r]

Facing this issue myself. Asked a question here: https://stackoverflow.com/questions/73679209/submitting-form-data-via-axios-fails-with-datasize-0