Storage hangs without completing or errors when uploading or getting files in parallel

[jellynoone]

Hi, @abhis3. I'm still experiencing this issue, although, I should add it doesn't happen all that consistantly. Here are the following specs: Device: MacBook Pro (Retina, 13-inch, Mid 2014) OS: macOS Big Sur, 11.7.8 Firebase tools: [email protected]

Additional context The error doesn't just happen when uploading many files but also when downloading many of them in parallel, for instance, in context of a messaging application this would be when downloading in-chat media, like images or videos.

On my end, if I try to download sometimes 30 or sometimes more images, but at around 60 the failure rate is almost consistant. At the same time, I get the following output in the command line, where the emulators are run:

{"result":{"permit":true},"id":82,"status":"ok"}
{"context":{"path":"/databases/(default)/documents/..."},"action":"fetch_firestore_document","server_request_id":83}

The images themselves are at average between 300kb - 500kb large.

I'm running into this problem when running a Flutter application, where a call writeToFile for the desired images, more or less so independent of one another i.e. it is coincidental that they are called in parallel as this depends on the images found on the particular screen.

Originally posted by @jellynoone in https://github.com/firebase/firebase-tools/issues/4918#issuecomment-1649195257

[jellynoone]

I'm reposting my comment in the linked issue since I'm still experiencing it. I'm also attaching my other responses to that issue in hopes of being helpful:

https://github.com/firebase/firebase-tools/issues/4918#issuecomment-1649205950 I also took a look at: https://github.com/firebase/firebase-tools/commit/dec8a3f0b2b3fa990da86af8b8d8926f8cf3d24e and I think I noticed a few things could be improved:

1. The test for this bug is incorrect
2. The number of test files being downloaded at the same time should probably be increased

Full reference to the test in question: https://github.com/firebase/firebase-tools/blob/8e3e2161b1da8e97bff8edbbe570d3dd15ec5f2a/scripts/storage-emulator-integration/conformance/firebase-js-sdk.test.ts#L492-L513

The specifically possibly incorrect part: https://github.com/firebase/firebase-tools/blob/8e3e2161b1da8e97bff8edbbe570d3dd15ec5f2a/scripts/storage-emulator-integration/conformance/firebase-js-sdk.test.ts#L502-L509

If we look at the test case, we see that the call to getDownloadURL is actually sequential because we are awaiting the await page.evaluate call. Meaning, we already get the result back before issuing a new getDownloadUrl.

I think the values retrieval here in the test should be rewritten as:

        const promises: Promise<string>[] = [];
        for (const singleFileName of allFileNames) {
          values.push(page.evaluate((filename) => {
            return firebase.storage().ref(filename).getDownloadURL();
          }, singleFileName));
        }
        const values = Promise.all(promises);

https://github.com/firebase/firebase-tools/issues/4918#issuecomment-1649219727 I also have an additional question about the error itself. It seems that somewhere the emulator pipes the result to the wrong output i.e. to the terminal and not to the emulator. This then seems to deadlock other promises, since they are waiting for a previous operation to complete, and it is waiting for a result that was passed to a different consumer. Is this analysis correct?

[jellynoone]

I would also like to add that I'm using security rules. Perhaps the evaluation of those is what ends up delaying the response to the point of deadlock??

[christhompsongoogle]

Thank you @jellynoone for your detailed analysis here. I'll update the test and see if we can get more information on the failure mode here.

[christhompsongoogle]

I added a test in https://github.com/firebase/firebase-tools/pull/6209 but wasn't able to reproduce the issue, up to 200 files and 2MB in size.

@jellynoone if you launch the emulator with the --debug flag that might help get more information on this bug.

[jellynoone]

@christhompsongoogle Thank you for the update. I'm not entirely aware of how the page.evaluate call works but does it also evaluate more complex security rules? For instance, with the project I'm experiencing this issue, I make a call to the firestore emulator to get the information about the document and validate the user's write and access permissions.

If page.evaluate does not do this, I suspect this might be why the bug isn't hit. In other words, I think the evaluation of the rules is fast enough that it doesn't hit the timeout: https://github.com/firebase/firebase-tools/blob/1a78f6434bbeb42d9af048f4a89dd120b6342c34/src/emulator/storage/rules/runtime.ts#L274-L278 This is mostly just speculation on my part, as I'm not deeply aware of how the internals here work. But it seems to me that this might be the reason since the bug does not always occur, which speaks to me of a timing issue.

[jellynoone]

I added a test in #6209 but wasn't able to reproduce the issue, up to 200 files and 2MB in size.

@jellynoone if you launch the emulator with the --debug flag that might help get more information on this bug.

I'll test with the --debug flag and come back with the results.

[jellynoone]

I tried testing with the --debug flag and could not reproduce the issue, but I could still reproduce it with the regular run. Does the --debug flag also make all calls sequential and not parallel similar to how --inspect-functions does for the functions emulator? If that is the case then this might be the reason why I'm unable to reproduce the issue.

As noted in my first response, the log also includes: {"context":{"path":"/databases/(default)/documents/..."},"action":"fetch_firestore_document","server_request_id":297} Which I assume is a log about a firestore get call in the security rules evaluation.

[PedroReyes]

Hello @jellynoone , @abhis3 ,

📖 Problem description

I am also experiencing this issue but in my case it is much simpler to get it. However, it is not consistent all the time. My use case is simple:

1. One react component reading from the storage everytime there is an update in a document. The size of the file is irrelevant as it is a simple JSON file.
2. I create a playwright test that runs in parallel 3 tests which load the same React component. Sometimes Playwright is lucky and there is no problem but some other there is. If I run playwright with just one single worker there is no issue. This sounds to be a problem with parallelism requests at the Firebase Local Emulator managing multiple requests.

🚩 Flag solution 🤔

🚩 I tried to create a flag within the code so if there is any other instance trying to do a fetch I will stop the fetching but that does not work always.

async function fetchUsers(set: any) {
    logInfo("🔄 Fetching current employees . . .");

    console.log(`${useUsersStore.getState().users.length} users fetched from Firebase Storage`);
    // Check if fetching is in progress - this is to avoid infinite loops in Firestore Emulator
    // Also this helps to not make multiple calls to Firebase Storage in production
    if (useUsersStore.getState().fetchingInProgress) {
        logInfo("🔄🟥 Fetching current employees - Fetching in progress - Skipping");
        return;
    }

    // Set fetching in progress to true 🚩 - flag
    set({
        fetchingInProgress: true
    });

    logInfo("🔄🟩 Fetching current employees - Fetching in progress - Not skipping");


    // Get firebase user
    const firebaseUser = useUsersStore.getState().userSignedIn;

    // Check if the user is signed in
    if (!firebaseUser) {
        logInfo("🔄🟥 Fetching current employees - User not signed in - Skipping");
        return;
    }

    const DATA_PROFILE_FOR_UI = STORAGE.USERS_DOC_PATH;

    // Create a reference to the file we want to download
    const storageRef = ref(storage, DATA_PROFILE_FOR_UI);

    // @TODO: Make this SSR by getting the data from a callable firebase function
    // Get the download URL
    const downloadURL = await getDownloadURL(storageRef); // ⛔ here the code stops working not returning anything ⛔

    // Download JSON file from URL
    const response = await fetch(downloadURL /*,{ cache: "no-store" }*/);

    // ... here it is the code for managing the respponse, irrelevant

    // Update the state with the sorted list of users
    set({
        . . .
    });

    logInfo("✅ Current employees fetched and updated successfully!");

    // Set fetching in progress to false 🚩 - flag
    set({
        fetchingInProgress: false
    });
}

⛔ Fetch method gets stuck

⛔ The next line of code simply gets stuck and does not return anything.

// Download JSON file from URL
const response = await fetch(downloadURL /*,{ cache: "no-store" }*/);

😫 I don't know what else to do to control this so multiple fetchings do not collapse in the emulator. I run also the debugging but nothing shows up in terms of errors while processing the requests.

✅ Solution

✅ I am running with one single worker my playwright tests so all the tests pass successfully due to this caveat with the emulator.

Best regards, Pedro Reyes.

[benjaminpetersen]

Hi all, I took a stab at fixing this https://github.com/firebase/firebase-tools/pull/6615

The problem is just JSON.parse running on whatever the java subprocess spits out. So debug() statements in your storage rules and 2 objects simultaneously will never resolve as the parse fails.

[transfer1992]

I am still experiencing this issue in March 2025. I can see that the proposed MR from @benjaminpetersen was never merged. Is there a path forward?

[loewj-reekon]

This issue breaks local development for any app that relies on Firebase Storage. We're experiencing hangs on nearly every file upload during development, with requests stuck in a pending state until the emulator is restarted. The behavior makes the Storage emulator effectively unusable. Thanks for the hard work. Hoping this gets unblocked soon.

[CambodianCoder]

My team is encountering this same issue. Would be nice to have https://github.com/firebase/firebase-tools/pull/6615 from @benjaminpetersen merged if that fixes the issue. Looks like the latest comment there is requesting test cases. One might be able to get some test cases pretty quickly using an AI code editor if time/resources are a constraint.