FR: Make possible to upload a file larger than 2GB in size

[0xjei]

[REQUIRED] Describe your environment

• Operating System version: Ubuntu 20.04.4 LTS - 64 bits
• Browser version: Brave Version 1.41.100 - Chromium: 103.0.5060.134 (Official Build) (64-bit)
• Firebase SDK version: 9.9.2
• Firebase Product: storage

[REQUIRED] Describe the problem

I'm making a client-side command line interface in NodeJS. This CLI will be used by users (i.e., they authenticate and make calls to cloud functions and storage). Therefore, I could not take advantage of the Firebase Admin SDK for NodeJS. My application needs to handle large files (~40 GB). When I try to upload a file with more than 2 GB in size using the Firebase JS SDK upload methods (e.g., uploadBytes()), I bump into an error RangeError [ERR_FS_FILE_TOO_LARGE]: File size (9663759512) is greater than 2 GB due to buffer size limits in NodeJS V8 engine (about ~1Gb for 32-bit os / ~2Gb for 64-bit os).

The main concern is with upload methods of this SDK (e.g., uploadBytes() or uploadString()) that take a Blob | Buffer | Uint8Array parameter as input for the file (i.e., the contents read and temporarily stored in a Buffer). Therefore, it is impossible to upload files larger than 2 GB in this case; only 2 Gb can be held in the buffer at a time which makes this task impossible. I thought there should be a possibility (as in the Admin SDK) to use write streams as is done for reading files (getStream() method which returns a NodeJS.ReadableStream).

I hope I am not wrong at all and this will generate further discussions. What are your thoughts on this? Hope to hear from you soon!

Steps to reproduce:

• Configure your firebase app in the relevant code below (.env or hardcoded).
• Add your local and Cloud Storage file paths (hardcoded).
• Run the script. (nb. I'm using Typescript, so build -> run)

Relevant Code:

import { initializeApp } from "firebase/app"
import { getStorage, ref, uploadBytes } from "firebase/storage"
import { readFileSync } from "fs"

(async () => {

    // init Firebase application.
    const firebaseApp = initializeApp({
        apiKey: process.env.FIREBASE_API_KEY,
        authDomain: process.env.FIREBASE_AUTH_DOMAIN,
        projectId: process.env.FIREBASE_PROJECT_ID,
        storageBucket: process.env.FIREBASE_STORAGE_BUCKET,
        messagingSenderId: process.env.FIREBASE_MESSAGING_SENDER_ID,
        appId: process.env.FIREBASE_APP_ID,
        databaseURL: process.env.FIREBASE_FIRESTORE_DATABASE_URL
    })

    // get storage instance.
    const firebaseStorage = getStorage(firebaseApp)

    // paths.
    const localPath = `local/path/to/file.txt` // > 2 GBs in size.
    const storagePath = `cloud/storage/path/to/file.txt`

    // get storage path ref.
    const pathReference = ref(firebaseStorage, storagePath)

    // call upload bytes method.
    await uploadBytes(pathReference, readFileSync(localPath))
})();

When I run the code (after configuration) with a file ~9.7 GBs in size, I got this error:

yarn build && yarn start2
yarn run v1.22.18
$ tsc
Done in 2.58s.
yarn run v1.22.18
$ node dist/src/index2.js
node:internal/errors:464
    ErrorCaptureStackTrace(err);
    ^

RangeError [ERR_FS_FILE_TOO_LARGE]: File size (9663759512) is greater than 2 GB
    at new NodeError (node:internal/errors:371:5)
    at tryCreateBuffer (node:fs:419:13)
    at readFileSync (node:fs:464:14)
    at file:///home/user/Documents/Code/sample_storage_error/dist/src/index2.js:61:65
    at step (file:///home/user/Documents/Code/sample_storage_error/dist/src/index2.js:32:23)
    at Object.next (file:///home/user/Documents/Code/sample_storage_error/dist/src/index2.js:13:53)
    at file:///home/user/Documents/Code/sample_storage_error/dist/src/index2.js:7:71
    at new Promise (<anonymous>)
    at __awaiter (file:///home/user/Documents/Code/sample_storage_error/dist/src/index2.js:3:12)
    at file:///home/user/Documents/Code/sample_storage_error/dist/src/index2.js:41:23 {
  code: 'ERR_FS_FILE_TOO_LARGE'
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.