Application crashes with java.util.concurrent.TimeoutException io.objectbox.BoxStore.close:707

[avarughese-tripactions]

trafficstars

Is there an existing issue?

• [] I have searched existing issues

Build info

• ObjectBox version: 4.0.3
• OS: Android 13, 14 and 15
• Device/ABI/architecture: happening across various OEM's and models. Some samples is Pixel 8 pro, Pixel8, Pixel 6a, Open CPH2551 etc

Steps to reproduce

We cannot reproduce the issue so far. It's happening at random in production with the new version upgrade from 3.7.1 to 4.0.3. The majority cases are happening right at the start/initialization of Object Box. But there are cases where we are seeing this crash quite randomly while the app is running

Expected behavior

The app should not crash

Actual behavior

The app is mostly crashing right at startup and sometimes randomly while in middle of execution.

Code

Code

This is how we do the initialization 

try {
   
    val builder = MyObjectBox.builder()
    boxStore = builder.androidContext(context)
        .name(custom_name)
        .build()
} catch (error: DbException) {

   boxStore?.removeAllObjects()
    if(!retry){
		// try it one more time
    }
}

Logs, stack traces

java.util.concurrent.TimeoutException: io.objectbox.BoxStore.finalize() timed out after 10 seconds at io.objectbox.BoxStore.close(BoxStore:707) at io.objectbox.BoxStore.finalize(BoxStore:525) at java.lang.Daemons$FinalizerDaemon.doFinalize(Daemons.java:370) at java.lang.Daemons$FinalizerDaemon.processReference(Daemons.java:350) at java.lang.Daemons$FinalizerDaemon.runInternal(Daemons.java:322) at java.lang.Daemons$Daemon.run(Daemons.java:131) at java.lang.Thread.run(Thread.java:1012)

Logs

[Paste your logs here]

[greenrobot-team]

Thanks for reporting! See https://github.com/objectbox/objectbox-java/issues/1190#issuecomment-2382501514.

Note: I labeled this issue with "more info required" so it will auto-close in a few days if there are no follow-up comments.

[avarughese-tripactions]

can you please provide some pointers on what kind of info you are looking for?

[RobbWatershed]

Interesting! I've been analyzing that very same issue on my app for weeks without any clear clue either.

I can't completely blame ObjectBox v4 as I've been migrating all my code to Kotlin at the same time, but I can't rule it out either.

[RobbWatershed]

One more info : I've been extensively reviewing threaded Box operations and made sure to call BoxStore.closeThreadResources() on the same thread that did the operations, using a finally block. That didn't really help in my case.

[avarughese-tripactions]

We figured out the problem. So basically what we see is while creating the Boxstore object it checks if a file is already open. If it is open it creates a checker thread which calls the system.gc while acquiring lock to the openFiles object. It tries to run finalization by looping. Meanwhile if the gc gets invoked then close() function is called via finalize(). This GC thread tries to acquire the lock on openFiles object which is not available. They get in to a deadlock situation causing a timeout This is the call stack scenario: BoxStore constrauctor-> verifyNotAlreadyOpen-> isFileOpen -> checkerThread checkerThread -> isFileOpenSync-> locks openFiles GC thread -> Finalize -> Close -> waits to acquire lock on openFiles

we had a corner case in the code which might try to create a new instance while previous one is not closed hence triggering this. Although we think the architecture of ObjectBox creation might need a change to avoid this deadlock Also, it doesn't look like lib version specific. I will let the ObjectBox to comment

[RobbWatershed]

we had a corner case in the code which might try to create a new instance while previous one is not closed hence triggering this.

If you mean an instance of BoxStore, aren't we supposed to create it inside a Singleton?

[avarughese-tripactions]

Yes, it should be singleton but we had a bug and that's triggering this issue for us. That's what we think

[greenrobot-team]

@avarughese-tripactions Thank you very much for the analysis and details! From what I remember it sounds plausible and we will have a look!

Edit: for our reference, the internal issue for this is objectbox-java#240

[RobbWatershed]

Hello @greenrobot-team! Any update on that one since january?

My users are still regularly reporting related crashes.

Thanks in advance~

[greenrobot-team]

@RobbWatershed Not yet. As previously explained, a workaround to avoid the assumed deadlock might be to make sure that if your app creates a new BoxStore instance that any previous one has been explicitly closed. Do not let your code rely on a previous instance getting garbage collected.

[RobbWatershed]

I happen to have two different BoxStores (A and B) in the same app. They have a distinct lifecycle and do not share the same file (differentiated by using BoxStoreBuilder.name upon initialization).

Is that issue supposed to happen if BoxStore A tries to get a lock just before BoxStore B is being GC'ed? Or is it only linked to BoxStores using the same file?

[greenrobot-team]

@RobbWatershed This affects all instances as the issue is about obtaining a lock on a static member:

https://github.com/objectbox/objectbox-java/blob/c84c0822c6d8e50ad9008794a3aa5e9913bc4a83/objectbox-java/src/main/java/io/objectbox/BoxStore.java#L87

[greenrobot-team]

@RobbWatershed Actually you are right, the "checker thread" that will hold a lock on openFiles is only created when there is a not closed BoxStore instance using the same database directory.

So BoxStore.close() getting stuck at trying to lock openFiles should only happen when the BoxStore constructor was called with the same database directory.

Edit: but the advice remains: do not rely on garbage collection. Always explicitly close the BoxStore instance once done using it.

[greenrobot-team]

I finally had a closer look at this and could not find a situation that leads to a deadlock.

Edit: however, there is a related issue where finalization of an existing, not referenced, BoxStore is not triggered by the checker thread, but will cause a DbException by BoxStore.verifyNotAlreadyOpen. This will be fixed in a future release.

@avarughese-tripactions suggested the System.gc() and System.runFinalization() calls of the "checker thread" (created when trying to create a BoxStore with a database directory that is in use by another, not closed, instance) might block, but in my tests on an Android 15 emulator and on a Eclipse Adoptium JDK 21 JVM this appears to not be the case. BoxStore.close() called by the finalizer is not blocked from proceeding.

A simple test for Android can look like this:

public class App extends Application {

    private BoxStore boxStore;

    @Override
    public void onCreate() {
        super.onCreate();

        boxStore = MyObjectBox.builder().androidContext(App.this).build();
        boxStore = null; // DON'T DO THIS, USE boxStore.close() instead!

        // DOESN'T get stuck (or crash due to store already open) here, old store is finalized by checker thread
        // Edit: only once we release a fix, version 4.3.1 or older crash with a "still open" DbException here
        boxStore = MyObjectBox.builder().androidContext(App.this).build();
}

we had a corner case in the code which might try to create a new instance while previous one is not closed hence triggering this.

Not explicitly closing the old instance but relying on the finalizer to do it is likely the reason of why it times out. In recent versions, BoxStore.close() was changed to wait on any pending database operations. So depending on what database operations are done BoxStore.close() may take longer to run than before.

So to resolve situations like this, make sure your code always explicitly closes BoxStore (and waits on it to complete) before removing the last reference to it. This will make sure a finalizer thread can complete quickly as the BoxStore is already closed.

[RobbWatershed]

Managed to fix my issue thanks to your insight. No problem left on my side. Thanks a lot @greenrobot-team ❤️

[greenrobot-team]

Thanks for the update!

I labeled this issue with "more info required" so it will auto-close in a few days if there are no follow-up comments.

[github-actions[bot]]

Without additional information, we are unfortunately not sure how to resolve this issue. Therefore this issue has been automatically closed. Feel free to comment with additional details and we can re-open this issue.