Storage hangs without completing or errors when uploading or deleting files in parallel

[b0ot]

[REQUIRED] Environment info

firebase-tools: 11.7.0

Platform: Windows 10 Pro running Ubuntu via WSL2

[REQUIRED] Test case

As you start to increase the number of items attempted to be uploaded to the storage emulator you will increase the chances of that upload failing. It will fail will even relatively small numbers of parallel uploads if attempted enough time (e.g. 2 or 3 files). I believe I also had it fail with a single file but haven't been able to reproduce as the failure is much less likely.

For example, if you try to upload 5x files via an approach similar to the one below: The first time maybe 3/5 will upload next 2/5 next 4/5 next 5/5

[REQUIRED] Steps to reproduce

`

// Upload Issue

const uploadFiles = () => {
  let uploadTasks = [];
  for (let i = 0; i < files.value.length; i++) {
    let curFile = files.value[i];

    let storageRef = fb_ref(storage, "imgs/" + curFile.name);
    const uTask = uploadBytesResumable(storageRef, curFile);
    uploadTasks.push(uTask);
    // Listen for state changes, errors, and completion of the upload.
    uploadTasks[i].on(
      "state_changed",
      (snapshot) => {
        // Get task progress, including the number of bytes uploaded and the total number of bytes to be uploaded

        console.log("stage changed for ", i, snapshot);
        const progress = snapshot.bytesTransferred / snapshot.totalBytes;

        //Update Progress
        uploadProgress.value[i].percent = progress;
        uploadProgress.value[i].color = "green-2";
        if (i == 1) {
          uploadProgress.value[i].color = "purple-2";
        }

        const progressPercent = progress * 100;
        // console.log("Upload is " + progressPercent + "% done");
        switch (snapshot.state) {
          case "paused":
            console.log("Upload is paused");
            break;
          case "running":
            console.log("Upload is running");
            break;
        }
      },
      (error) => {
        console.log("failed ============================== ", error);
        switch (error.code) {
          case "storage/unknown":
            // Unknown error occurred, inspect error.serverResponse
            break;
        }
      },
      () => {
        // Upload completed successfully, now we can get the download URL
        //Update Progress
        console.log("Trying to get url for foo: ", i);
        getDownloadURL(uploadTasks[i].snapshot.ref)
          .then((downloadURL) => {
            console.log("File Download foo: " + i, downloadURL);
          })
          .catch((e) => {
            console.log("error foo: ", e);
          });
      }
    );
  }
  Promise.all(uploadTasks).then(() => {
    console.log("Files done uploading");
  });`

// Delete Issue
  try {
    let deletePromises = [];
    console.log("deleting storage ", imgArray);
    imgArray.forEach((img) => {
      let imgPath = `${img.path}${img.name}`;
      let deleteRef = ref(storage, imgPath);
      console.log("path: ", imgPath);
      deletePromises.push(deleteObject(deleteRef));
    });
    await Promise.all(deletePromises);
  } catch (e) {
    console.log("Error: ", e);
  } finally {
    Loading.hide();
  }

[REQUIRED] Expected behavior

It should be possible to consistently upload or delete multiple files in parallel to storage

[REQUIRED] Actual behavior

The process will randomly hang without showing any errors. It just never completes.

[Gremash]

I experienced the same issue on MacOS with firebase-tools 11.8.0

[cdnicoll]

I'm seeing a similar issue only with getDownloadUrl() Using the emulators and looping through image references, it hangs on the promise.

const imagesMapped = []
  for (const imagePath of arrayOfImagesPath) {
    const storageRef = ref(storage, imagePath)
    try {
      const storageUrl = await getDownloadURL(storageRef)
      imagesMapped.push(storageUrl)
    } catch (e) {
      console.error(e)
      throw e
    }
  }

Doesn't appear to be a staging or production issue

[abhis3]

Hi @b0ot and @cdnicoll, can you try running this against the latest version of firebase-tools and seeing if the issue is still there? There was a fix released in v11.8.0 that fixed what I believe is the issue you're running into. See https://github.com/firebase/firebase-tools/issues/3915 for the issue that I believe is similar.

[abhis3]

@Gremash I know you're on 11.8.0, mind upgrading your firebase-tools version and reporting back? Curious to see the code as well that you're running and getting some more diagnostic details as well so we can repro it on our side.

[cdnicoll]

@abhis3 Just tested with 11.9.0 and it still appears to hang. Pulling from emulators appear to be a bit better but still not great

[callum-gander]

Also experiencing this issue with

"firebase": "^9.9.3",
"firebase-tools": "^10.9.2",

My code looks something like this

 const uploadFiles = await Promise.all(
                filesToUpload.map(async (file) => {
                    // * set file's

                    const uploadTask = await uploadBytesResumable(
                        file.ref,
                        file.file
                    );

                    return uploadTask;
                })
            );
            console.log(uploadFiles);
            await Promise.all(
                filesToUpload.map(async (file) => {
                    const url = await getDownloadURL(file.ref);
                    fileURLs[`${file.name}`] = url;
                    return;
                })
            );
            console.log(uploadFiles);

It either works perfectly or will just randomly hang for no reason and require repeatedly restarting my requests until firebase decides not to hang. It's been a really big issue getting our uploads consistent due to this. It's been an issue with both the emulator and the live version. Any potential fixes or things to look for? I can't upgrade at the moment, see #6621

[Rexios80]

I'm seeing the same issue. I get lines like this in the console and the storage emulator stops responding.

{"result":{"permit":false},"id":18,"status":"ok"}
{"result":{"permit":false},"id":19,"status":"ok"}

I'm calling getDownloadUrl to fetch multiple images from storage at once. It doesn't break every time, but it's often enough that the storage emulator is basically useless.

[abhis3]

@cdnicoll Interesting thanks for the information. I'm not able to reproduce this locally but I have a theory that I'd like your help testing if that's alright. Do you mind changing the value here in your local version of the package and retesting?

You can follow the steps outlined here, which should guide you through this process. Maybe bumping it to ~50 or so and seeing if that helps? If so please let me know and we can work together to find a more suitable lock duration for everyone.

In the meantime, I'll try another couple things to reproduce this on my end. Pipes exhibit different behaviors on different systems so mind telling me what OS and Platform you're running your code on?

[abhis3]

@callum-gander Mind dropping a link to #6621, I can't find it on my end. You're seeing this in production as well as emulator? My first recommendation would be to upgrade to the latest version of firebase-tools and trying again at least as part of the emulator fix, as we've had some changes go out that should alleviate the issues you're experiencing

[abhis3]

@Rexios80 Please follow the recommendation of this comment https://github.com/firebase/firebase-tools/issues/4918#issuecomment-1242381562, the issue you're encountering is identical to #3915 and has been closed out

[Rexios80]

The emulator is running on macOS, the app using the emulator is running in an iOS simulator. I have the latest version of firebase-tools installed (I update when prompted)

[abhis3]

@Rexios80 Mind sharing your exact version of firebase-tools and a code snippet of how you're calling getDownloadUrl multiple times at once? Is it via a Promise.all?

[Rexios80]

rexios ~ % firebase --version
11.14.3

I'm using dart so:

/// Get the download urls for all files at the given [path]
static Future<List<String>> getDownloadUrls(String path) async {
  final result = await _storage.ref(path).listAll();
    final urls = <String>[];
    for (final ref in result.items) {
      final url = await ref.getDownloadURL();
      urls.add(url);
    }
    return urls;
}

[abhis3]

@Rexios80 Thanks! I'll start taking a look into this. A possible temporary solution to unblock you could be what's outlined here and bumping the delay lock timeout manually. Curious to see if bumping this up higher to ~50ms or so would fix the issue, if so then we need to reevaluate our lock timeout number.

I'll also in the meanwhile do some digging into a way to circumvent this annoying pipe behavior and hopefully squash this issue for good.

[cdnicoll]

@abhis3 Hey, just ran through the setup process and verified using npm ls -g --depth=0 --link=true that it was linked.

.nvm/versions/node/v16.16.0/lib
└── [email protected] -> ./../../../../../gitrepo/firebase-tools

I made the change found here and watched the build via npm run build:watch

Running my project I didn't notice a huge difference. Images do load, but it can take several seconds of waiting for them to load in.

I tried putting in a few debug statements (console.logs) within the firebase tools, but didn't see output - wondering if theres logs or anything for that?

Let me know next steps, happy to continue testing out changes

[callum-gander]

@callum-gander Mind dropping a link to #6621, I can't find it on my end. You're seeing this in production as well as emulator? My first recommendation would be to upgrade to the latest version of firebase-tools and trying again at least as part of the emulator fix, as we've had some changes go out that should alleviate the issues you're experiencing

Apologies, I meant #6621 the firebase-js-sdk repo https://github.com/firebase/firebase-js-sdk/issues/6621. I'll update to see if it's still breaking and post here later

[zachcaldwell]

+1 to all of this.

I'm on MacOS Monterey, [email protected] and am having issues using getDownloadURL.

[abhis3]

@zachcaldwell thanks for the info, mind sharing some of the specs of your machine you're running this on? I suspect this may be a machine specific issue as I'm not able to repro this on my end on my MacBook. I have an alternative fix in mind that I'll be working on to hopefully alleviate this for all users

[abhis3]

@cdnicoll Thanks for doing that! Can I ask what value you changed the 15 to? The higher the value goes, the higher the delay will be for images to load in but the more time we give the pipe buffer to clear. I'm wondering if there's a slightly higher value than 15 that might work more for a broader set of users while I work on the longer term fix

[abhis3]

Closing as I'm not able to repro this on my end. Please feel free to reopen with your machine specs if this is still an issue.

[jellynoone]

Hi, @abhis3. I'm still experiencing this issue, although, I should add it doesn't happen all that consistantly. Here are the following specs: Device: MacBook Pro (Retina, 13-inch, Mid 2014) OS: macOS Big Sur, 11.7.8 Firebase tools: [email protected]

Additional context The error doesn't just happen when uploading many files but also when downloading many of them in parallel, for instance, in context of a messaging application this would be when downloading in-chat media, like images or videos.

On my end, if I try to download sometimes 30 or sometimes more images, but at around 60 the failure rate is almost consistant. At the same time, I get the following output in the command line, where the emulators are run:

{"result":{"permit":true},"id":82,"status":"ok"}
{"context":{"path":"/databases/(default)/documents/..."},"action":"fetch_firestore_document","server_request_id":83}

The images themselves are at average between 300kb - 500kb large.

I'm running into this problem when running a Flutter application, where a call writeToFile for the desired images, more or less so independent of one another i.e. it is coincidental that they are called in parallel as this depends on the images found on the particular screen.

[jellynoone]

I also took a look at: https://github.com/firebase/firebase-tools/commit/dec8a3f0b2b3fa990da86af8b8d8926f8cf3d24e and I think I noticed a few things could be improved:

1. The test for this bug is incorrect
2. The number of test files being downloaded at the same time should probably be increased

Full reference to the test in question: https://github.com/firebase/firebase-tools/blob/8e3e2161b1da8e97bff8edbbe570d3dd15ec5f2a/scripts/storage-emulator-integration/conformance/firebase-js-sdk.test.ts#L492-L513

The specifically possibly incorrect part: https://github.com/firebase/firebase-tools/blob/8e3e2161b1da8e97bff8edbbe570d3dd15ec5f2a/scripts/storage-emulator-integration/conformance/firebase-js-sdk.test.ts#L502-L509

If we look at the test case, we see that the call to getDownloadURL is actually sequential because we are awaiting the await page.evaluate call. Meaning, we already get the result back before issuing a new getDownloadUrl.

I think the values retrieval here in the test should be rewritten as:

        const promises: Promise<string>[] = [];
        for (const singleFileName of allFileNames) {
          values.push(page.evaluate((filename) => {
            return firebase.storage().ref(filename).getDownloadURL();
          }, singleFileName));
        }
        const values = Promise.all(promises);

[jellynoone]

I also have an additional question about the error itself. It seems that somewhere the emulator pipes the result to the wrong output i.e. to the terminal and not to the emulator. This then seems to deadlock other promises, since they are waiting for a previous operation to complete, and it is waiting for a result that was passed to a different consumer. Is this analysis correct?

[zigzagph]

I have been running into a very similar issue.

Device: MacBook Pro M2 Pro OS: Sonoma 14.4.1 firebase-tools: 13.0.3

My issue involves uploading a number of (10 - 25) files to a local storage emulator. Sometimes files would upload and sometime they would not. It was very hit or miss and impossible to debug. The error I was seeing in the emulator output is below.

{"result":{"permit":true},"id":117,"status":"ok"} {"result":{"permit":true},"id":120,"status":"ok"} {"result":{"permit":true},"id":121,"status":"ok"} {"result":{"permit":true},"id":125,"status":"ok"} {"result":{"permit":true},"id":126,"status":"ok"}

After updating the timeout from 15ms to 60ms I am still seeing the issue. This error is present when using either the uploadBytes or uploadBytesResumable methods to upload.

Thank you for all your help and suggestions.