firebase-tools icon indicating copy to clipboard operation
firebase-tools copied to clipboard
verified publisher icon firebase

Storage Emulator Hangs with Parallel Uploads (4+ Concurrent Requests)

Open luckyape opened this issue 1 year ago • 4 comments

[REQUIRED] Environment info

  • firebase-tools: 13.4.1
  • Platform: macOS Sonoma 14.3.1

[REQUIRED] Test case

      const media = [jpg1, jpg2, jpg3, jpg4]; // files range in size from 1mb to 1.5mb
      const uploadPromises = media.map((file) => {

        const storageRef = ref(storage, `temp/${file.path}`);
        return await uploadBytes(storageRef, file);
      });
      await Promise.all(uploadPromises);
      console.log('Upload promises complete');

[REQUIRED] Steps to reproduce

Run code with firebase storage and emulators configured. Observe that only the first two images upload successfully.

[REQUIRED] Expected behavior

4 images should be uploaded to storage. Promises should resolve and 'Upload promises complete' should be logged to the console. Synchronous execution of the uploads does work.

[REQUIRED] Actual behavior

Only 2 images are uploaded to storage. Promises hang and do not resolve.

[debug] [2024-03-10T20:18:06.822Z] Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead INFO: Detected non-HTTP/2 connection. {"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}} [debug] [2024-03-10T20:18:06.854Z] Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead INFO: Detected non-HTTP/2 connection. {"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}} [debug] [2024-03-10T20:18:06.856Z] Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead INFO: Detected non-HTTP/2 connection. {"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}} [debug] [2024-03-10T20:18:06.918Z] Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead INFO: Detected non-HTTP/2 connection. {"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}} [debug] [2024-03-10T20:18:07.159Z] Mar. 10, 2024 5:18:07 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead INFO: Detected non-HTTP/2 connection. {"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:07 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}} [debug] [2024-03-10T20:18:07.216Z] Mar. 10, 2024 5:18:07 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead INFO: Detected non-HTTP/2 connection. {"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:07 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}} [info] {"result":{"permit":true},"id":15,"status":"ok"} {"result":{"permit":true},"id":16,"status":"ok"} {"metadata":{"emulator":{"name":"storage"},"message":"{"result":{"permit":true},"id":15,"status":"ok"}\n{"result":{"permit":true},"id":16,"status":"ok"}"}} [debug] [2024-03-10T20:18:07.947Z] >>> [apiv2][query] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast [none] [debug] [2024-03-10T20:18:07.947Z] >>> [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"eventId":"1710101887947","timestamp":"2024-03-10T20:18:07.947Z","eventType":"google.storage.object.finalize","resource":{"service":"storage.googleapis.com","name":"projects//buckets/hellograham.appspot.com/objects/temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg","type":"storage#object"},"data":{"kind":"storage#object","name":"temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg","bucket":"hellograham.appspot.com","generation":"1710101887908","metageneration":"1","contentType":"image/jpeg","timeCreated":"2024-03-10T20:18:07.908Z","updated":"2024-03-10T20:18:07.908Z","storageClass":"STANDARD","size":"1133688","md5Hash":"6XXm9KGgFk9ge8Pl/YYnpw==","etag":"6TEW5Thl0dy3G0IbWd9RgR8D3iA","metadata":{"firebaseStorageDownloadTokens":"906fda07-0dd7-4c49-85b5-0edb4e36d9b4"},"crc32c":"igflCw==","timeStorageClassUpdated":"2024-03-10T20:18:07.908Z","id":"hellograham.appspot.com/temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg/1710101887908","selfLink":"http://127.0.0.1:9199/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F3be4fc5f-d678-467e-bce7-655566f2d091.jpg","mediaLink":"http://127.0.0.1:9199/download/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F3be4fc5f-d678-467e-bce7-655566f2d091.jpg?generation=1710101887908&alt=media"}} [debug] [2024-03-10T20:18:07.951Z] <<< [apiv2][status] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast 200 [debug] [2024-03-10T20:18:07.951Z] <<< [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"status":"multicast_acknowledged"} [debug] [2024-03-10T20:18:07.952Z] >>> [apiv2][query] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast [none] [debug] [2024-03-10T20:18:07.952Z] >>> [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"specversion":"1.0","id":"338f9c11-a6ab-4b3e-9895-b073b28fcbb2","type":"google.cloud.storage.object.v1.finalized","source":"//storage.googleapis.com/projects//buckets/hellograham.appspot.com/objects/temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg","time":"2024-03-10T20:18:07.908Z","data":{"kind":"storage#object","name":"temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg","bucket":"hellograham.appspot.com","generation":"1710101887908","metageneration":"1","contentType":"image/jpeg","timeCreated":"2024-03-10T20:18:07.908Z","updated":"2024-03-10T20:18:07.908Z","storageClass":"STANDARD","size":"1133688","md5Hash":"6XXm9KGgFk9ge8Pl/YYnpw==","etag":"6TEW5Thl0dy3G0IbWd9RgR8D3iA","metadata":{"firebaseStorageDownloadTokens":"906fda07-0dd7-4c49-85b5-0edb4e36d9b4"},"crc32c":"igflCw==","timeStorageClassUpdated":"2024-03-10T20:18:07.908Z","id":"hellograham.appspot.com/temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg/1710101887908","selfLink":"http://127.0.0.1:9199/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F3be4fc5f-d678-467e-bce7-655566f2d091.jpg","mediaLink":"http://127.0.0.1:9199/download/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F3be4fc5f-d678-467e-bce7-655566f2d091.jpg?generation=1710101887908&alt=media"}} [debug] [2024-03-10T20:18:07.955Z] <<< [apiv2][status] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast 200 [debug] [2024-03-10T20:18:07.955Z] <<< [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"status":"multicast_acknowledged"} [debug] [2024-03-10T20:18:07.959Z] >>> [apiv2][query] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast [none] [debug] [2024-03-10T20:18:07.959Z] >>> [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"eventId":"1710101887959","timestamp":"2024-03-10T20:18:07.959Z","eventType":"google.storage.object.finalize","resource":{"service":"storage.googleapis.com","name":"projects//buckets/hellograham.appspot.com/objects/temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","type":"storage#object"},"data":{"kind":"storage#object","name":"temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","bucket":"hellograham.appspot.com","generation":"1710101887923","metageneration":"1","contentType":"image/jpeg","timeCreated":"2024-03-10T20:18:07.923Z","updated":"2024-03-10T20:18:07.923Z","storageClass":"STANDARD","size":"1515111","md5Hash":"KyqvqiFDwGtMrSnrGyXyYg==","etag":"y2qK0V857ty6e0G93oORBd4dtlc","metadata":{"firebaseStorageDownloadTokens":"a5c9469a-acd9-4e37-8115-bcca4c0028ed"},"crc32c":"w2JVsA==","timeStorageClassUpdated":"2024-03-10T20:18:07.923Z","id":"hellograham.appspot.com/temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg/1710101887923","selfLink":"http://127.0.0.1:9199/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","mediaLink":"http://127.0.0.1:9199/download/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg?generation=1710101887923&alt=media"}} [debug] [2024-03-10T20:18:07.963Z] <<< [apiv2][status] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast 200 [debug] [2024-03-10T20:18:07.963Z] <<< [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"status":"multicast_acknowledged"} [debug] [2024-03-10T20:18:07.963Z] >>> [apiv2][query] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast [none] [debug] [2024-03-10T20:18:07.963Z] >>> [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"specversion":"1.0","id":"9692e1ed-b450-4b40-a74f-4e1db1dbeae7","type":"google.cloud.storage.object.v1.finalized","source":"//storage.googleapis.com/projects//buckets/hellograham.appspot.com/objects/temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","time":"2024-03-10T20:18:07.923Z","data":{"kind":"storage#object","name":"temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","bucket":"hellograham.appspot.com","generation":"1710101887923","metageneration":"1","contentType":"image/jpeg","timeCreated":"2024-03-10T20:18:07.923Z","updated":"2024-03-10T20:18:07.923Z","storageClass":"STANDARD","size":"1515111","md5Hash":"KyqvqiFDwGtMrSnrGyXyYg==","etag":"y2qK0V857ty6e0G93oORBd4dtlc","metadata":{"firebaseStorageDownloadTokens":"a5c9469a-acd9-4e37-8115-bcca4c0028ed"},"crc32c":"w2JVsA==","timeStorageClassUpdated":"2024-03-10T20:18:07.923Z","id":"hellograham.appspot.com/temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg/1710101887923","selfLink":"http://127.0.0.1:9199/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","mediaLink":"http://127.0.0.1:9199/download/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg?generation=1710101887923&alt=media"}} [debug] [2024-03-10T20:18:07.965Z] <<< [apiv2][status] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast 200 [debug] [2024-03-10T20:18:07.965Z] <<< [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"status":"multicast_acknowledged"}

luckyape avatar Mar 10 '24 20:03 luckyape

Hey @luckyape, thanks for reaching out and for providing a detailed report and description of the issue you’re encountering. I’m currently working on replicating this, but I’m unable to reproduce the same behavior. When uploading 4 files (~1MB in size each) at the same time, all of them get uploaded without any issues to the Storage emulator.

I created this repo to try and replicate this. Could you let me know if you’re able to reproduce the issue using the steps provided, or if I’m missing anything to reproduce the issue? Also, if you can provide a minimal reproducible example, please share it so that we can replicate the issue on our end.

aalej avatar Mar 12 '24 18:03 aalej

@aalej Thanks again for your help. I have been able to recreate the error in the React Admin context. https://github.com/luckyape/issue-6865-react-admin

On Wed, Mar 13, 2024 at 6:18 AM graham charles @.***> wrote:

for now I take it back...seems there is still something going on, looping back on trying to isolate the issue

On Tue, Mar 12, 2024 at 7:05 PM graham charles @.***> wrote:

Thanks for the quick response. The error was on my end, as I was returning an await storagePromise rather than just the promise.

If you would like to see how to produce the error behavior for some reason: https://github.com/luckyape/issue-6865

I will update the ticket incase anyone else is trying to debug the same issue.

graham

On Tue, Mar 12, 2024 at 3:26 PM aalej @.***> wrote:

Hey @luckyape https://github.com/luckyape, thanks for reaching out and for providing a detailed report and description of the issue you’re encountering. I’m currently working on replicating this, but I’m unable to reproduce the same behavior. When uploading 4 files (~1MB in size each) at the same time, all of them get uploaded without any issues to the Storage emulator.

I created this repo https://github.com/aalej/issues-6865 to try and replicate this. Could you let me know if you’re able to reproduce the issue using the steps provided, or if I’m missing anything to reproduce the issue? Also, if you can provide a minimal reproducible example https://stackoverflow.com/help/minimal-reproducible-example, please share it so that we can replicate the issue on our end.

— Reply to this email directly, view it on GitHub https://github.com/firebase/firebase-tools/issues/6865#issuecomment-1992293078, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGB2UEYY4XQIEU3JSGU2TTYX5CFPAVCNFSM6AAAAABEPH4N5OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJSGI4TGMBXHA . You are receiving this because you were mentioned.Message ID: @.***>

-- Please lmk if you have any questions. g

-- Please lmk if you have any questions. g

-- Please lmk if you have any questions. g

luckyape avatar Mar 13 '24 10:03 luckyape

Thanks @luckyape for sharing a repo for this. I tried using the application you shared, but I’m still unable to reproduce the issue. The steps I took were:

  1. Run firebase emulators:start --project project_id
  2. Run yarn to install dependencies
  3. Add Firebase configs to https://github.com/luckyape/issue-6865-react-admin/blob/6a9695dc58ed9c4bf55ea29b10dbb6084cf4511d/src/App.tsx#L7-L9
  4. Run yarn dev to start the app
  5. Open "http://localhost:5173/#/firebase/create"
  6. Select files to upload
  7. Click the “UPLOAD” button
  8. Checking the Storage emulator shows that the files were uploaded: storage-em-ss

I think I’m probably missing something here to reproduce the issue. To make sure that we're on the same page, could you let me know if the steps I took to reproduce the issue are correct?

aalej avatar Mar 15 '24 13:03 aalej

i have some other tasks prioritized at the moment, but i will quickly add for now that the behavior does upload some of the images before failing. ensure the console is clean and perhaps test with additional images. I promise to loop back and try to get a more detailed response by the beginning of next week. thanks again for your help @aalej

On Fri, Mar 15, 2024 at 10:39 AM aalej @.***> wrote:

Thanks @luckyape https://github.com/luckyape for sharing a repo for this. I tried using the application you shared, but I’m still unable to reproduce the issue. The steps I took were:

  1. Run firebase emulators:start --project project_id
  2. Run yarn to install dependencies
  3. Add Firebase configs to https://github.com/luckyape/issue-6865-react-admin/blob/6a9695dc58ed9c4bf55ea29b10dbb6084cf4511d/src/App.tsx#L7-L9
  4. Run yarn dev to start the app
  5. Open "http://localhost:5173/#/firebase/create"
  6. Select files to upload
  7. Click the “UPLOAD” button
  8. Checking the Storage emulator shows that the files were uploaded:

storage-em-ss.png (view on web) https://github.com/firebase/firebase-tools/assets/118378103/7456217b-afe6-4f0c-b955-0ad3d37f96a4

I think I’m probably missing something here to reproduce the issue. To make sure that we're on the same page, could you let me know if the steps I took to reproduce the issue are correct?

— Reply to this email directly, view it on GitHub https://github.com/firebase/firebase-tools/issues/6865#issuecomment-1999687765, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGB2UD4FOSA6F7XLZK5AODYYL2ZZAVCNFSM6AAAAABEPH4N5OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJZGY4DONZWGU . You are receiving this because you were mentioned.Message ID: @.***>

-- Please lmk if you have any questions. g

luckyape avatar Mar 15 '24 14:03 luckyape

I apologize for not getting to this when I said I would. I've created a screencast of the error. https://www.youtube.com/watch?v=ZPv8sjU9tcg

If I can help in any way please let me know.

On Fri, Mar 15, 2024 at 11:15 AM graham charles @.***> wrote:

i have some other tasks prioritized at the moment, but i will quickly add for now that the behavior does upload some of the images before failing. ensure the console is clean and perhaps test with additional images. I promise to loop back and try to get a more detailed response by the beginning of next week. thanks again for your help @aalej

On Fri, Mar 15, 2024 at 10:39 AM aalej @.***> wrote:

Thanks @luckyape https://github.com/luckyape for sharing a repo for this. I tried using the application you shared, but I’m still unable to reproduce the issue. The steps I took were:

  1. Run firebase emulators:start --project project_id
  2. Run yarn to install dependencies
  3. Add Firebase configs to https://github.com/luckyape/issue-6865-react-admin/blob/6a9695dc58ed9c4bf55ea29b10dbb6084cf4511d/src/App.tsx#L7-L9
  4. Run yarn dev to start the app
  5. Open "http://localhost:5173/#/firebase/create"
  6. Select files to upload
  7. Click the “UPLOAD” button
  8. Checking the Storage emulator shows that the files were uploaded:

storage-em-ss.png (view on web) https://github.com/firebase/firebase-tools/assets/118378103/7456217b-afe6-4f0c-b955-0ad3d37f96a4

I think I’m probably missing something here to reproduce the issue. To make sure that we're on the same page, could you let me know if the steps I took to reproduce the issue are correct?

— Reply to this email directly, view it on GitHub https://github.com/firebase/firebase-tools/issues/6865#issuecomment-1999687765, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGB2UD4FOSA6F7XLZK5AODYYL2ZZAVCNFSM6AAAAABEPH4N5OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJZGY4DONZWGU . You are receiving this because you were mentioned.Message ID: @.***>

-- Please lmk if you have any questions. g

-- Please lmk if you have any questions. g

luckyape avatar Mar 23 '24 16:03 luckyape

Hey @luckyape, thanks for all the information you shared. It is highly appreciated! I am now able to reproduce the issue. A few things to note, to repro the issue I did the ff:

  1. Increase the size of the images to ~2mb each. I can’t seem to reproduce the issue when using smaller images
  2. Increase the number of images being uploaded to around 6 at the same time.
  3. Upload multiple times. From what I can tell, the initial upload works just fine, all images are uploaded. However, succeeding uploads will only upload a few images (4/6 images would get uploaded)

Let me raise this to our engineering team so that they can take a look and provide some context on what could be causing this issue.

aalej avatar Mar 27 '24 13:03 aalej