firebase-tools
firebase-tools copied to clipboard
firebase
firestore emulator ui: some collections will not show data, error in console
Some collections will not show data, and will produce an error in the browser console. If I click the collection name in the emulator ui, it will show the doc id of the first document in the URL, but will show no content.
Interestingly my app using the emulator works perfectly fine, and I can even use firebase-admin to query the data, but if I click any of the affected collections, it shows it as if it was empty. Other collections work just fine, and show data as expected.
The total import folder is roughly 12 MB, which I would not consider a large dataset.
[REQUIRED] Environment info
firebase-tools: 11.0.0
Platform: macOS Monterey 12.3.1 (21E258) (M1 chip)
[REQUIRED] Test case
As this appears to be somewhat specific to data, I doubt it could be reproduced at will easily, but it is true that some of my other imports with similar structure do not exhibit this error.
[REQUIRED] Steps to reproduce
run this:
firebase --debug emulators:start --only auth,firestore,functions --import=./path/to/import
Then go to the firestore emulator in the browser, and click one of the affected collection names:
http://XXX.XXX.XXX.XXX:4000/firestore/data/<affected_collection>
[REQUIRED] Expected behavior
Document list and their content should show in the ui.
[REQUIRED] Actual behavior
Document list and their content is not showing in the ui, only the + Add document, and to the right the + Start collection and + Add field options, and an error in the browser console:
react_devtools_backend.js:4026 [2022-05-20T08:14:57.612Z] @firebase/firestore: Firestore (8.8.1): Connection WebChannel transport errored: sr {type: 'c', target: or, g: or, defaultPrevented: false, status: 1}
...and...
GET http://XXX.XXX.XXX.XXX:8080/google.firestore.v1.Firestore/Listen/channel?database=projects%2Fproject-name%2Fdatabases%2F(default)&VER=8&RID=rpc&SID=XXXXXXXXX%3D%3D&CI=0&AID=0&TYPE=xmlhttp&zx=XXXXXXXXXX&t=1 400 (Bad Request)
Relevant part of the output of the logs with --debug flag:
May 20, 2022 10:20:09 AM com.google.net.webchannel.server.v8.BackChannel add
WARNING: Failed to send a new message due to too many pending messagings in the back channel (10001). May need enable flow control.
May 20, 2022 10:20:09 AM com.google.net.webchannel.server.v8.ChannelInternalImpl handle
WARNING: Server fails to send the message, abort the channel!
May 20, 2022 10:20:09 AM com.google.cloud.datastore.emulator.firestore.webchannel.FirestoreV1WebChannelAdapter$FirestoreListenHandler onError
WARNING: [ERROR] NETWORK_ERROR ()
Is there anything special with the affected collection names? For example, are they longer, do they contain special characters, contain more documents etc.? How many documents are we working with?
@yuchenshi no, the collection names are relatively short, and only contain English alphabet characters camelCased if slightly longer, the affected onces are called 'bookings', and 'bookingsMeta', and they are called the same in the dataset where there is no error (I'm trying to work off of backups of a live system).
First I thought it had something to do with the firebase-export-metadata.json, maybe the version number in it, but I made a 1:1 copy of it from a working dataset and that did not help. Then I thought maybe it was the number of accounts (was a total of 502, though I don't consider that to be a lot), so I reduced it to less than 100 - no difference.
The not so funny thing about it is that it showed no problem for a very long time, I had worked with several different ones of these backups and mostly just ran admin queries on them to analyse the data, some of them I haven't even touched at all, and then it just suddenly started happening with these relatively bigger ones, but again, they're not huge or anything, their total size isn't more than 12 MB on disk (the working one is < 1MB), the affected collections are definitely the two biggest ones compared to the ones that work.
It literally happened between two queries I ran to loop through a larger part of the dataset, before the query I could still search documents in those two collections, after it I started to get these errors, and no data shown.
@yuchenshi
I have the same issue. firebase-tools: 11.0.1 Platform: Ubuntu 20.04.4 LTS
Collection name is 'emissions'
Added info: if I know the document id, and either type it in in the address bar as part of the query string, or in the emulator UI on top of the collection list at the end of the path, and then reload (sometimes a couple of times), it will show the document details then, but only then.
Hi all, thanks for the information. Look like this wasn't a known problem (e.g. identifier format or special characters) and we'll gladly help digging into this. Would any of you mind providing the export dump used? If it consists of sensitive data, consider stripping them (as long as the issue can still be reproduced), or sending it privately to Firebase Support.
@yuchenshi I'll try to create a masked data dump for you; it may take a couple of days (at least?), but I am very much willing to help in the interest of all, and of course myself :)
(oops, sorry for triggering the Needs: Attention flag unnecessarily...)
@yuchenshi I now have a masked copy of the dataset I use in my emulator. How should I share it?
@benomatis Thanks! See if you can just upload it here. If not, please share via Google Drive and I'll route it to my team.
Thanks for uploading the export. With that, I've successfully reproduced the issue and I've passed it on internally to the relevant teams. We are unable to promise any timeline for this, but if others also have this issue, adding a +1 on this issue can help us prioritize adding this to the roadmap.
(Googler-only internal tracking bug: b/236770121)
Also, I've isolated the issue to the Firestore emulator -- the issue can be reproduced even without the UI. However, the problem does not happen with non-realtime requests, such as curl 'http://localhost:8080/v1/projects/demo-example/databases/(default)/documents/bookings'.
Therefore, a possible workaround would be exporting all relevant data using REST API / Node.js admin / Firestore JS SDK Lite and recreating the database from scratch. We understand that this is not ideal, but this may unblock you while we're working on it. If you notice that this issue reoccurs while you're rebuilding the database, please also let us know about the details. Observations like "this starts happening as I add this particular document" can go a long way on helping us pinpointing the issue.
@yuchenshi I don't completely understand the workaround, can you please elaborate? I use firestore, not realtime database.
@benomatis Sure! The mentioned workarounds are of course all about Firestore (not RTDB). Which part would you like further elaboration?
@yuchenshi How do I recreate the database from scratch? I mostly use node scripts with firebase admin that I subsequently run from the command line when I have specific data to extract - so do you mean I create a node script like that, for example, export the data into, say, an array, delete it from firestore, then re-import it from said array?
so do you mean I create a node script like that, for example, export the data into, say, an array, delete it from firestore, then re-import it from said array?
Yes, that's pretty much it. You can also save the data somewhere else (e.g. in a file), restart the emulator (with no import) and continue the script from a clean start if that helps.
Thanks @yuchenshi I may give that workaround a try if it becomes a serious problem.
Hi firebase team,
@yuchenshi is there any status update on this issue, or at least some timeline? I am having the same problem where for a collection which has too many documents, the local emulator does not show the documents. My import is 29.3 MiB. The data is there since its displayed on the client side, but it cannot be accessed in the emulator
+1
I have 2 MiB import data, I can't see any collection's document, my collections have short name.
Notes:
- One of my team member can see the documents and everything looks fine, he is using Windows.
- All MacOs including M1 can't see the documents
- We tried java version: 11 and 18.
Same issue here, we used to be able to run dumps from our staging and prod projects to run some tests locally, since a few months ago we can't the data is not large enough (a few MB of data), we have a maximum of 3 level deep sub collections, and no weird names are used in collections or documents. The data is there because we can query from code, we just can't see it in the UI, and we see all the errors in the dev console listed here.
+1
Since a few days ago no prod dump we make works on the emulator, it hangs loading forever
In order to keep this bug as focused as possible I removed some comments that were off topic. Thanks all for your patience.
Apologies; I've been trying to keep the conversation where you indicate you want it, not "get more exposure" for my questions (my intent was moving my unanswered questions so you didn't have to). Where do you want discussion to continue on this?
Hey @benomatis (and others following this issue reporting similar errors), an FYI for you: there was a changed pushed in firebase-tools v11.25.3 that might help with this particular error. You may want to try updating to the 11.25.3 version and see if that solves this issue for you.
@MidnightLightning I can confirm v11.25.3 fixed this bug / error. Thank you so much!