Firestore query failed with code=RESOURCE_EXHAUSTED, description=Quota exceeded.

[thomasdao]

Describe your environment

• Android Studio version: Arctic Fox 2020.3.1 Patch 3
• Firebase Component: Firestore
• Component version:

implementation platform('com.google.firebase:firebase-bom:28.4.1')
implementation 'com.google.firebase:firebase-firestore-ktx'

[REQUIRED] Step 3: Describe the problem

My app queries from Firestore, and when the result contains more than 2000 items, the query always fails with error WatchStream]: (c98c54b) Stream closed with status: Status{code=RESOURCE_EXHAUSTED, description=Quota exceeded., cause=null}. The detail error stacktrace can be downloaded here.

Note:

• This problem only happens on Android SDK. I am able to query on iOS and Web without a problem.
• This problem happens when each row contains quite significant amount of data. In my test, the final result contains around 5000 items, total size is about 120MB
• I am on Blaze plan
• I have waited for more than 30 minutes before executing a new query

Steps to reproduce:

Simply execute a query. The query will fail for user with a lot of data.

Relevant Code:

fun getAll() {
    val ref = FirebaseFirestore.getInstance()
        .collection("users")
        .document(userId)
        .collection("notes")
        .limit(2500)

    // This query always failed with the limit >= 2500
    ref.get(Source.SERVER)
        .addOnSuccessListener { results ->
            Log.d("Test","results size" + results.size())
        }.addOnFailureListener {
            it.printStackTrace()
            Log.d("Test", "exception ${it.localizedMessage}")
        }
}

[aguatno]

Hi @thomasdao thanks for reporting. One possible reason that this happens is because you might be crossing a limit that exist in Firestore regardless of plan. The system might return “resource exhausted” in various different scenarios.

I would recommend you accessing the Google Cloud Platform console to get a more accurate depiction of the usage your database is experiencing. I suggest this as sometimes the Firebase Console isn't as accurate as the GCP one.

[thomasdao]

@aguatno thanks for replying. As I mentioned in the original report, the query is successful for iOS and Web Firestore SDK, this problem only happens for Android. I don't think this problem is due to the limit (otherwise the iOS and Web query should have failed).

[thebrianchen]

@thomasdao This seems like a backend issue, since the SDK only retries and surfaces the errors. Can you file a ticket via Firebase support channel so the backend team can take a look at your project's logs?

[thomasdao]

@thebrianchen I contacted support - the backend team did not see any error with "RESOUCE_EXHAUSTED" in my project log.

Francisco at support team has been able to reproduce this issue and find out the problem is due to the way Firestore Android SDK handle the cache. If the cache folder is deleted when the query returns, the exception does not happen, however I hope the Firestore Android team could confirm if this is indeed the root case.

[thebrianchen]

@thomasdao Yes, after looking at Francisco's report + the absence of backend logs, it's clear that the error is thrown by SQLite after the cache is full. To workaround this, you can either clear the cache directory with Android utility methods, or increase the size of the cache via Firestore settings.

Internally, we'll look at improving documentation around this error case and improving the error messages. I'm going to close the issue, but feel free to ask any other questions you may have!

[thomasdao]

@thebrianchen I would prefer to keep this issue open since it is a quite serious bug and affect a lot of our users. The work around to clear the cache is only a work around - we don't know if it is a real root cause, and if it is the root cause, it would be better to fix on the SDK side. I think "improving documentation and error message" is not a correct approach to this problem.

[thomasdao]

Update: when I apply the work around today - it no longer works! It looks like the work around is not a real root cause to this problem and this problem still happens regularly for me.

[thomasdao]

@thebrianchen can you reopen this issue? This is serious bug and I don’t understand why you are so eager to close it without even investigating the issue

[bacarPereira]

In my case I exceeded the amount of written documents / day in firestore (Spark plan)

.

[thebrianchen]

@thomasdao The "Quota exceeded" error is thrown by SQLite when the cache is full, and not by the backend. We have an internal bug b/201092032 investigating your issue, and we created a repro that verified it was a cache issue.

Which workaround did you try? The initial cache size in Android is set to 100MB, so you could go over the limit in a single query. If you hit the cache size limit, you'll either have to experiment with periodically clearing your cache or increasing the cache limit. You can also remove the cache size limit (documentation) to see if this solves your issue.

I'm keeping the issue closed since this is working as intended. I'm happy to answer any other questions or investigate a custom repro scenario that you may have.

[thomasdao]

@thebrianchen I have set the cache size to unlimited, the problem still happens:

fun start() {
    val fireStore = FirebaseFirestore.getInstance()

    val settings = firestoreSettings {
        isPersistenceEnabled = true
        cacheSizeBytes = FirebaseFirestoreSettings.CACHE_SIZE_UNLIMITED
    }

    fireStore.firestoreSettings = settings
}

Everytime when I started the query and when the query completed (both successful or failure), I clear the cache:

App.shared.cacheDir.delete()

The firebase dependency is the latest implementation platform('com.google.firebase:firebase-bom:29.0.0')

The error log is the same like before:

2021-11-30 10:05:24.576 9233-9353/com.getupnote.android W/Firestore: (24.0.0) [WatchStream]: (94af14f) Stream closed with status: Status{code=RESOURCE_EXHAUSTED, description=Quota exceeded., cause=null}.
2021-11-30 10:05:34.585 9233-9233/com.getupnote.android W/System.err: com.google.firebase.firestore.FirebaseFirestoreException: Failed to get documents from server. (However, these documents may exist in the local cache. Run again without setting source to SERVER to retrieve the cached documents.)
2021-11-30 10:05:34.586 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.Query.lambda$getViaSnapshotListener$1(Query.java:986)
2021-11-30 10:05:34.586 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.Query$$ExternalSyntheticLambda1.onEvent(Unknown Source:8)
2021-11-30 10:05:34.586 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.Query.lambda$addSnapshotListenerInternal$2$com-google-firebase-firestore-Query(Query.java:1133)
2021-11-30 10:05:34.586 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.Query$$ExternalSyntheticLambda2.onEvent(Unknown Source:6)
2021-11-30 10:05:34.586 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.core.AsyncEventListener.lambda$onEvent$0$com-google-firebase-firestore-core-AsyncEventListener(AsyncEventListener.java:42)
2021-11-30 10:05:34.586 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.core.AsyncEventListener$$ExternalSyntheticLambda0.run(Unknown Source:6)
2021-11-30 10:05:34.586 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.util.Executors$$ExternalSyntheticLambda0.execute(Unknown Source:0)
2021-11-30 10:05:34.587 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.core.AsyncEventListener.onEvent(AsyncEventListener.java:39)
2021-11-30 10:05:34.587 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.core.QueryListener.raiseInitialEvent(QueryListener.java:176)
2021-11-30 10:05:34.587 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.core.QueryListener.onOnlineStateChanged(QueryListener.java:116)
2021-11-30 10:05:34.587 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.core.EventManager.handleOnlineStateChange(EventManager.java:178)
2021-11-30 10:05:34.587 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.core.SyncEngine.handleOnlineStateChange(SyncEngine.java:368)
2021-11-30 10:05:34.587 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.core.MemoryComponentProvider$RemoteStoreCallback.handleOnlineStateChange(MemoryComponentProvider.java:119)
2021-11-30 10:05:34.587 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.remote.RemoteStore$$ExternalSyntheticLambda0.handleOnlineStateChange(Unknown Source:2)
2021-11-30 10:05:34.588 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.remote.OnlineStateTracker.setAndBroadcastState(OnlineStateTracker.java:181)
2021-11-30 10:05:34.588 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.remote.OnlineStateTracker.lambda$handleWatchStreamStart$0$com-google-firebase-firestore-remote-OnlineStateTracker(OnlineStateTracker.java:121)
2021-11-30 10:05:34.588 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.remote.OnlineStateTracker$$ExternalSyntheticLambda0.run(Unknown Source:2)
2021-11-30 10:05:34.588 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.util.AsyncQueue$DelayedTask.handleDelayElapsed(AsyncQueue.java:144)
2021-11-30 10:05:34.588 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.util.AsyncQueue$DelayedTask.$r8$lambda$9BXioagktdupZWFTAaXUZGAu7Gk(Unknown Source:0)
2021-11-30 10:05:34.588 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.util.AsyncQueue$DelayedTask$$ExternalSyntheticLambda0.run(Unknown Source:2)
2021-11-30 10:05:34.588 9233-9233/com.getupnote.android W/System.err:     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:462)
2021-11-30 10:05:34.589 9233-9233/com.getupnote.android W/System.err:     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2021-11-30 10:05:34.589 9233-9233/com.getupnote.android W/System.err:     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
2021-11-30 10:05:34.589 9233-9233/com.getupnote.android W/System.err:     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
2021-11-30 10:05:34.589 9233-9233/com.getupnote.android W/System.err:     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
2021-11-30 10:05:34.589 9233-9233/com.getupnote.android W/System.err:     at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor$DelayedStartFactory.run(AsyncQueue.java:234)
2021-11-30 10:05:34.589 9233-9233/com.getupnote.android W/System.err:     at java.lang.Thread.run(Thread.java:923)

[thomasdao]

@bacarPereira that's strange, I thought 20,000 writes per day is only the free quota and not for the Blaze Plan?

[thebrianchen]

@thomasdao Can you provide a repro app? Someone from our team was able to reproduce the same error you're getting by filling up the cache with 1MB documents (see repo). By deleting the clearing the cache in between queries, they were able to make the error go away.

I'll investigate making the error message less opaque, but in the meantime, getting a repro is likely the only feasible way to make progress on your issue since the error message are the same.

[bacarPereira]

@bacarPereira that's strange, I thought 20,000 writes per day is only the free quota and not for the Blaze Plan?

That's right. I meant spark plan. I will edit. Thanks.

[thomasdao]

@thebrianchen this is easily reproducible on my sample project. Please let me know how to send to you, thanks

[thebrianchen]

@thomasdao If the repo is public, can you paste a link here? Alternatively, you can add me onto the repo.

[thomasdao]

@thebrianchen thanks, I've added you to the private repo on Github. Please let me know if you need other info, thanks

[thebrianchen]

@thomasdao Thanks, will take a look tomorrow and investigate.

EDIT: I have a few other things on my plate, but I'll get to it this week!

[thebrianchen]

Reopening since a repro was provided.

[larssn]

Wondering what the status of this one is? You guys seemed pretty enthusiastic about it initially. 🙂

Ran into it myself a few days ago, but from Flutter running on Android. Not sure how to reproduce it, but it happened when listening to a collection that returned a fair amount of data. Big enough to trigger the garbage collector in Android for a while. While GC was running, the underlying framework threw those resource exhausted errors.

The end result was that several snapshot listeners died, without getting any more updates from the backend.

Sorry, not much to go by.

[wu-hui]

Previous owners of this ticket no longer work on Firestore.

@thomasdao Are you still seeing this issue? Do you have your repro handy still?

[thomasdao]

@wu-hui I ended up disable the cache instead of increasing the cache size, and this issue does not happen anymore. I think this can be closed, those who see this problem can just disable the cache.

[larssn]

Disabling the cache isn't a fix. Please reopen

[iocuydi]

Also happening for web version when cache is disabled

[larssn]

@thebrianchen thanks, I've added you to the private repo on Github. Please let me know if you need other info, thanks

@thomasdao Do you still have the private repo you talked about? I'm just interested in an actual dev getting some insight on this.

[iocuydi]

could this potentially be related to firewall settings? I have a web app using firestore (Javascript sdk) that has functioned fine for nearly a year, today some of our customers had their corporate issue laptops replaced and now they all get this issue. Every other firebase product (auth, db, storage, functions) works fine. The app doesn't use caching (offline persistence) and never has.

[thomasdao]

@larssn Sorry I can't share the private repo since it contains real data. This issue is not related to firewall settings, in my test it always happens.

Beside from disabling the cache, I need to paginate the data and only load about 100 documents at a time, this probably depends on the size of the document. For me, the document size varies but usually between 100KB up to 1MB, and loading around 2000 documents at a time usually cause memory problems.

[larssn]

Is any work being done on this? At least how to surface a better error message than "Resource exhausted", which is just too little to go by.

[ChiragMoradiya]

We have encountered the same issue.

For us, it's being reproduced most of the time, when we run 5 concurrent queries expecting 1000-1500 documents in each. We use Firestore Javascript SDK to integrate on a webapp.

See
and Video for reference.

Please, note that it doesn't happen if we run a single query to retrieve 10K documents.

Other observations:

1. Once this error appears, snapshot handler is invoked with 0 docs.
2. And, Firestore Javascript Library seems to be performing an internal retry automatically, but after delay of a minute or more; and in most cases it also fails.
3. It may succeed after 2-3 such retry attempts, each with a delay of 1 to 2 minutes. So, the query response is retrieved after ~10 minutes.

We are on the Blaze Plan. I didn't find any documentation about this error: Why it appears? What should be standard practices to avoid this?

[antonimso]

Has anyone else found a solution to the problem? the same error appears to me