uploading large file 3GB or more as readablestream consume huge amount of memory.

[Guling85]

Im trying to upload a large file around 3GB with TUS client. It seems to me that TUS client doesnt use readablestream as intended because the memory usage is huge. The memory usage of chrome is like 2.5GB for 3 GB file.

here is my code.

 const file = $event.target.files[0];

      const upload = new Upload(file.stream().getReader(), {
        endpoint: "https://localhost:8443/upload/file",
        retryDelays: [0, 3000, 6000, 12000, 24000],
        chunkSize: 6553600,
        uploadLengthDeferred: true,
        metadata: {
          uuid: result[0].uploadId,
          order: chunkOrder.toString()
        },
        //chunkSize: 1048576,
        onBeforeRequest: (req) => {
          const xhr = req.getUnderlyingObject();
          xhr.withCredentials = false;
        },
        onError: async (error) => {
          console.log(error);

        },
        onProgress: function (bytesUploaded, bytesTotal) {
          let percentage = Math.ceil(bytesUploaded / bytesTotal * 100);

          console.log(bytesUploaded, bytesTotal, percentage + "%")
        },
        onSuccess: async () => {
          console.log('Upload success');
        }
      });

      upload.start();

Im a doing this wrong?

Because when using "file" instead of "file.stream()" works fine. The reason i dont wanna use "file" is because im gonna make a custom ReadableStream that modifies the data and upload it as a stream.

[Acconut]

Thank you for this report. Does the memory consumption also grow in Firefox or browsers other than Chrome?

[Guling85]

Hi same in Firefox and edge.

[Acconut]

Thanks, I will look into this when I have the time.

[ISnowFoxI]

Hello, any update on this?

[Acconut]

I tested a similar setup as @Guling85 described:

new Upload(file.stream().getReader(), {
        chunkSize: 6553600,
        uploadLengthDeferred: true,

However, I did not find any signs of a memory leak or excessive memory usage in general when uploading a 1GB file. In Chrome, the heap size was around 17MB, while Firefox only reported a 3MB heap (which seems a bit odd given that we configured a buffer of 6553600 bytes).

@ISnowFoxI Are you experiencing memory issues with tus-js-client in the browser? If not, I will close this issue. Since it was originally opened, we also reworked parts of the client for handling stream resource. So it is possible that there was an issue in older versions.

[ISnowFoxI]

@Acconut No, I was using it on react native (Ios and Android), and it seems like it was because of Blob, that said, I am still not sure what the issue was.

[Acconut]

Ok, feel free to open another issues if you experience issues with tus-js-client on React Native again. I will close this ticket because the original problems is not present anymore.