Firestore LevelDbDocumentOverlayCache infinite recursion results in broken app on launch

[sergiocampama]

Description

It appears that there is a weird issue in firestore which results in an overlay proto message that can't be decoded correctly by nanopb, and ends up in an infinite recursion which ultimately crashes the app. You can see the infinite recursion in the provided crash report.

Copilot-2022-11-08-152134.ips.zip

This first happened on FirebaseFirestore 10.0.0, and nanopb on commit 819d0a2, tag 0.3.9.9 (this was the version of nanopb that Cocoapods installed as a dependency of Firebase). I jumped from Firestore 9.1.0 to 10.0.0. In 9.1.0 I had never seen this issue, and in 10.0.0 I've already had 2 reports on beta testers, which makes me wary of releasing an update with 10.0.0 to the public.

I was able to dump the offending proto message just before the infinite recursion, which I provide below: nanopbbreakingmessage.bin.zip . This message is of type google_firestore_v1_Write.

If you notice, that file is pretty big, considering that it has 2 user provided fields for update, and a bunch of repeated __previous_value__, __local_write_time__, __type__ and serv er_timestamp fields. I tried reproducing this error by creating a sample test suite in nanopb to read this message, but on nanopb@master I wasn't able to replicate, and going back to commit nanopb@819d0a2 I wasn't able to run it because of development environment issues.

Reproducing the issue

(Continued from above since there's a limit)

I am at a loss as to how to continue to debug, given the firestore model of sending work to a work queue, making it almost impossible to find where the work is being queued from. I also can't do a binary search to try and find which version introduced this issue (given the overlay mutation is stored somewhere on disk, so even if I rollback to an older version of Firestore it will continue to crash). For some reason reverting back to 9.1.0 avoids the issue, but any version above that presents it (tried 9.6.0 and 9.4.0... 9.3.0 and 9.2.0 have another crashing bug so those versions are invalid for us)

I have the problematic ldb files to share if they are useful, but don't want to share them publicly as it may have some sensitive data.

I have to say, every time I update Firestore I find some issue that appears randomly and forces me to stay in an older version. It's getting incredibly frustrating to maintain a consumer app in these conditions.

(Back to Reproducing the issue)

I have a setup that reproduces the issue somewhat reliably, but I can't share that given it's a private project with private settings, but I'd be happy to jump on a call and show what's happening.

Firebase SDK Version

10.0.0

Xcode Version

14.0

Installation Method

CocoaPods

Firebase Product(s)

Analytics, Authentication, Crashlytics, DynamicLinks, Firestore, Functions, Messaging, Remote Config, Storage

Targeted Platforms

iOS, macCatalyst

Relevant Log Output

See crash report above.

If using Swift Package Manager, the project's Package.resolved

No response

If using CocoaPods, the project's Podfile.lock

Can't add it since "it's too long".

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[ehsannas]

Thanks for reporting @sergiocampama. I will start looking into this. In the meantime, if there has been any developments over the past few weeks, please provide an update here. Thanks

[sergiocampama]

not much to report given we're pinned to Firestore 9.1.0

[ehsannas]

Thanks @sergiocampama . I'm sorry to hear about the inconvenience. I've been trying to find meaningful pointers to look into from the crash report, but there's nothing that can point to the right direction. The content of nanopbbreakingmessage.bin also don't seem right.

Do you currently have a way to reproduce the issue that you can share with us?

I'm also curious whether you can try a newer version (e.g. 10.3.0) and let us know if you experience the issue in the latest version as well?

[google-oss-bot]

Hey @sergiocampama. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[sergiocampama]

Not sure how much more data I can provide, I can't reproduce this consistently, it just happens from time to time

[sergiocampama]

Thanks @ehsannas for looking into this, I am testing the newer versions of firestore and will report back if we still see this issue

[sergiocampama]

We haven't found this issue again after 2 weeks, I'd consider this one closed.

[rizafran]

Thanks for the update, @sergiocampama. I'll now be closing this ticket.