File upload got stuck while uploading large text files

[SamantaChinmoy]

Hi,

I am using multer to upload a zip file which contains large text files. Zip file size is 960 MB. There are total 10 text files is there inside zipped file, size of each text file is in the range of 500-2 gb. While I am uploading this zipped file using multer , The file upload got stuck and it is not processed further after 919 mb. But I have tested with other zipped file with size of 1.5 gb which contains many small size text file in the range of 1-2 mb, that time it is got successfully uploaded.

N.B: I am using disk based storage.

[almigiusz]

@SamantaChinmoy have you solved it? I have the same problem.

[aroldoGoulart-estuda]

Same problem here

[joeyguerra]

have ya'll tried the latest release candidate? npm install [email protected]? It adds a stream property on the req.file so you can pipe directly to disk. I would try that and see what you get.

[johnny77221]

I have same issue uploading a video file It works well when I run the code on Mac but the file does not receive completely on deployed Ubuntu

tried 2.0.0-rc.2 and this happens https://github.com/nodejs/node/issues/38058 for 2.0.0 rc.3 and rc.4, I cannot even import mutler: Uncaught Error Error [ERR_REQUIRE_ESM]: require() of ES Module /(project_path)/node_modules/multer/index.js from /(project_path)/index.js not supported.

but I am using node v21.1.0, is it necessary to downgrade to v15.3 as mentioned above?

[joeyguerra]

can you provide any code? test script? Technically, the ERR_REQUIRE_ESM error is because the consuming code is trying to require an ESM module.

[johnny77221]

I am building with nodejs v21 package.json:

"dependencies": {
    "body-parser": "^1.20.2",
    "cookie-parser": "^1.4.6",
    "dotenv": "^16.4.5",
    "express": "^4.19.1",
    "express-fileupload": "^1.5.0",
    "multer-utf8": "^1.4.5-lts.11",
    "socket.io": "^4.7.5",
    "uuid": "^9.0.1"
  }

code:

const express = require('express')
const multer = require('multer-utf8')
const storage = multer.diskStorage({
  destination: function (request, file, callback) {
      callback(null, './temp/')
  },
  filename: function (request, file, callback) {
      callback(null, file.originalname)
  }
})
const app = express()

...

app.post('/api/upload-video',multer({ storage: storage }).single('videoFile'), async (req, res) => {
 if (!req.file ) {
      return res.status(400).send('no file uploaded')
  }
  // process logics
  return res.status(201).send('{"message": "created"}')
})

[joeyguerra]

Have you looked into nodes request timeout as noted on https://github.com/mscdex/busboy

[johnny77221]

after lots of try and error I found this way working for 8GB file

nginx sites-available conf:

server {
    server_name (my DN);
    location / {
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://localhost:3000;
        client_max_body_size 20480M;

        proxy_request_buffering off;
        client_body_timeout 1200s;   // 3200s for 15GB file
        proxy_send_timeout 3600;  // 9600 for 15GB file
        proxy_read_timeout 3600;  // 9600 for 15GB file
    }
}

node js app code:

const multer = require('multer-utf8')
const storage = multer.diskStorage({
  destination: function (request, file, callback) {
      callback(null, './temp/')
  },
  filename: function (request, file, callback) {
      callback(null, file.originalname)
  }
})
const app = express()
const http = require('http').createServer(app)

app.post('/api/upload-video',multer({ storage: storage, limits: { fileSize: 21474836480 /* 20GB */ } }).single('videoFile'), async (req, res) => {
  if (!req.file ) {
      return res.status(400).send('no file uploaded')
  }
  req.setTimeout(86400000)
  // process logics
  return res.status(201).send('{"message": "created"}')
})

so basically my issue is mainly related by nginx forwarding request, the timeout and file size limit for the mutler also affects