[Flutter] `openStore()` is not working across multiple FlutterEngines

[navaronbracke]

ℹ️ Quick links:

• There is an explanation
• There is a workaround

(Edit by ObjectBox)

---

I have a use case where I have a Flutter app that does two things.

• a void main(){} entrypoint that runs a regular Flutter app (i.e. runApp(MaterialApp())) This app uses the database normally. It also schedules tasks using the workmanager plugin
• the workmanager plugin that executes the tasks that the app schedules

The problem

The void main(){} entrypoint is executed from the default FlutterEngine (created by the Activity/AppDelegate) The workmanager plugin creates a new FlutterEngine for each task it needs to run. This is because the DartExecutor from the original FlutterEngine is executing the void main(){} entrypoint. A DartExecutor can only run one entrypoint at a time.

Because there are two FlutterEngines (each with their own Isolate pool and such), the Dart code is not in sync between the Engines. That is, one engine might have a Store _myStore that is not null, but the other engine still has one that is null (because it does not have the same memory pool allocated).

This results in the following code failing on the FlutterEngine that didn't open the store:

class MyDbService {
  static Store _myStore;

  Future<void> init() async {
   // The store is null in the other entrypoint, since that is an entire new block of memory, owned by another engine.
   // This results in the error
   // Bad state: failed to create store: 10001 Cannot open store: another store is still open using the same path 
   _store ??= await openStore();
  }
}

Describe the solution you'd like

I'd like to be able to use a Store across different FlutterEngines. If openStore() would return a new connection, regardless of the FlutterEngine, that would be sufficient I think. (I.e. using a connection pool) I'd expect the ObjectBox API to work as-is through this connection. I.e. CRUD / observable queries should work on this new connection.

Then I could do something like this to fix my problem:

main.dart

void main() async {
   await DbService().initialize(); // Open the database service in the first FlutterEngine

  runApp(MyApp()); // Does call to `WorkManager.registerTask()` which invokes the task runner function
}

my_app.dart

class MyApp extends StatefulWidget {
 // ...
}

class _MyAppState extends State<MyApp> {
 @override
 Widget build(BuildContext context){
    return Scaffold(
      appBar: AppBar(title: Text('My app')),
      body: Center(
        child: ElevatedButton(
           child: Text('Schedule task'),
           onPressed: (){
             WorkManager().registerTask('MyTask', {'input': 'foo'});
           }
        ),  
      ),
    );
  }


  @override
  void dispose() async {
   await DbService().close(); //Close the database service in the first FlutterEngine
   super.dispose();
  }
}

task_runner.dart

@pragma("vm:entry-point")
void _taskRunner(){
   WidgetsFlutterBinding.ensureInitialized();

    Workmanager().executeTask((taskName, inputData) async {
      await DbService().initialize(); // open a new connection in this FlutterEngine

      switch(taskName){
        case 'MyTask':
          // do work
          break;
      }

      await DbService().close(); // open the connection that was closed in this FlutterEngine
    }
}

Additional note

This could also benefit the add-to-app use case where people use a FlutterEngine per view they embed into an existing native app.

[greenrobot-team]

Thanks for reporting!

First of all, using multiple Flutter Engines seems to be a valid use case. The API is advertised to be used to have one or more Flutter screens/views in an otherwise non-Flutter app as the note above says.

I'm not familiar with how this API works, so the following might not be right: the ObjectBox store is created via FFI and refered to using a native pointer. So it should technically be accessible from anything that runs in the same Dart VM. So to access an open store attach to it using Store.attach?

[navaronbracke]

@greenrobot-team I could try to use Store.attach() in the handler that is run on the second FlutterEngine. However, since the second FlutterEngine is created by a background thread (started natively by WorkManager), I think I'll end up with a second Dart VM, which won't see the initialized store from the first one (or the other way around). I'll give it a try and let you know.

[navaronbracke]

@greenrobot-team I had another shot at it and Store.attach() did not work. I used the following code:

  Future<String> _getDatabaseDirectoryPath() async {
    final dir = await getApplicationDocumentsDirectory();

    // The default object box dir is inside the application documents dir,
    // under the `/objectbox` folder.
    return '${dir.path}${Platform.pathSeparator}objectbox';
  }

  Future<void> initialize() async {
    final dbPath = await _getDatabaseDirectoryPath();

    try {
      // Try to open the store normally.
      _store ??= await openStore(directory: dbPath);
    } catch (error) {
      // If the store cannot be opened, it might already be open.
      // Try to attach to it instead.
      _store = Store.attach(getObjectBoxModel(), dbPath);
    }
  }

In the background thread I never end up in the Store.attach() phase since the Store is null in that Isolate (because it runs on a different FlutterEngine and thus does not share memory with the first FlutterEngine). Only in the first Isolate the Store is not null. This results in the openStore() function throwing

Bad state: failed to create store: 10001 Cannot open store: another store is still open using the same path

I think I need a way to check if the native store is open (internally that should check the FFI pointer) and then I could use Store.attach()? It should be possible since the native store throws that state error?

I'll be happy to provide a minimal reproducible sample app to pinpoint the problem.

[navaronbracke]

@greenrobot-team In relation to the other issue I had, I'll try to check if the store is open with that static method. Maybe that fixes this issue?

[navaronbracke]

@greenrobot-team I got it working using Store.isOpen() and using attach if its already open. I have one more question though: Does the Store emit database updates to each connection? I.e. if I make changes in connection 2, will connection 1 be able to see them? My use case is that the background worker modifies the database and the app observes those changes through its own connection.

[greenrobot-team]

I read through the code example and docs from Flutter: there should only exist a single Dart VM for all FlutterEngines. And yes, they do not share state.

So _store ??= doesn't work. Using Store.isOpen(path) and then calling attach instead of open as you mentioned is the the way to go then.

Change notifications should happen on any isolate/engine as the native side is handling notifications. E.g. it should be possible to put an object in a background worker which is observed by a watched query in the UI.

Edit: let me know if this resolves your original request (and this can be closed).

[navaronbracke]

@greenrobot-team This does indeed resolve my problem, thank you. And yes I managed to use Store.isOpen() to fix the connection issue.

Closing as working as intended.

[animedev1]

@navaronbracke navaronbracke Hi Can you tell me how did you get your Workmanager works with ObjectBox ? I had tried these two and couldn't open the database in background-isolates that's why I changed part or my database into Drift which works but not perfect as sometimes it throws error about database being locked. I wanna give Objectbox another try if it support multi-isolates Thank you in advance

[animedev1]

Confirmed work! I had migrated all the drift code into objectbox which works perfectly fine and no more database locked I also removed drift package

[madrojudi]

Hello! @navaronbracke and @animedev1, please, can you tell me how you did it?

This is my code to init my store

if(Store.isOpen(null)) {
        print("first attach");
        _store = Store.attach(getObjectBoxModel(), null);
        print("first attach done");
      } else {
        try {
          print("try open");
          _store = await openStore();
          print("try open done");
        }catch (ex) {
          print("catch to open $ex");
        }
      }

When my application is not closed, but it is not in the foreground, it generate this error, using Firebase Cloud Messaging on background : Bad state: failed to create store: 10001 Cannot open store: another store is still open using the same path: "/data/user/999/io.artcreativity.app/app_flutter/objectbox"

When app is closed, it work well.

You can learn more about my issue here #451

Thank you!

[navaronbracke]

@madrojudi I did it like this:

    if (Store.isOpen(dbPath)) {
      _store = Store.attach(getObjectBoxModel(), dbPath);

      return;
    }

    _store = await openStore(directory: dbPath);

_store is a variable of type Store? which I use to store the opened store. dbPath is the path to the database as specified by attach & openStore, but you probably have that already.

[madrojudi]

Thank you @navaronbracke Unfortunately it doesn't always work for me. But I found an alternative that is currently working.

1. I check if DB is open with Store.isOpen(path). If it is true, I use Store.attach
2. If Store.isOpen is false, I try to open new Store by openStore. When it open, I save the reference into a file
3. If Store.isOpen throw error, I check to read the reference which I save in 2. and I use Store.fromReference to open Store

Currently, It is working. I will continue testing to see if it is stable.

[meomap]

Thank you madrojudi

I tried you way and it works sometimes. Other times, Saving reference then reading back will throw error: [ERROR:flutter/lib/ui/ui_dart_state.cc(198)] Unhandled Exception: Invalid argument(s): Reference.processId 692 doesn't match the current process PID 2046

As in our case, we fire background process when users interact with Widget remote views so the current workaround will eventually lead to race condition

[angus-clark]

When I try to call Store.isOpen inside the Workmanager isolate it always returns false even if the store is actually already open. Not sure why this is happening??

[greenrobot-team]

@clarky2233 This should typically not happen as Store.isOpen is calling into the C library that has shared state across isolates. Can you open a new issue with more details, e.g. the Flutter/Dart version you are using?

[angus-clark]

I was passing the incorrect path to the function, it seems to work now. As a follow up, is it expected to only work when passing a specific path rather than null?

[greenrobot-team]

@clarky2233 Calling Store.isOpen(null) will check the default path, which is objectbox. This will not work for Flutter apps as there a path in the documents directory is used when opening a Store.

We should update the docs on how to get the correct default path for Flutter. Edit: done.

[greenrobot-team]

As an alternative to an open check and then doing either open or attach, see https://github.com/objectbox/objectbox-dart/issues/442#issuecomment-1464909888 for a workaround on Android.

According to this comment having multiple engines is a more common occurrence than thought (e.g. when deep-linking or opening from a notification), so maybe we should update the docs and maybe even offer API for this.

[techouse]

@greenrobot-team please also mention in the docs how to properly get the default path of the ObjectBox store in Flutter, i.e.

import 'package:path/path.dart' as path;
import 'package:path_provider/path_provider.dart';

Future<void> main() async {
  final String storeDirectoryPath = path.join(
   (await getApplicationDocumentsDirectory()).path,
   Store.defaultDirectoryPath,
  );
}

[greenrobot-team]

@techouse Thanks, as mentioned above already have done this. It just wasn't released, yet. https://github.com/objectbox/objectbox-dart/blob/fe91dbf782e94d0493f4d0e48f79e75614b81cab/objectbox/lib/src/native/store.dart#L446-L454

Edit: this was included with release 2.0.0.

[user97116]

Logs

E/UIFirst (12492): failed to open /proc/12492/stuck_info, No such file or directory
E/UIFirst (12492): failed to open /proc/12492/stuck_info, No such file or directory
Reloaded 1 of 3008 libraries in 4,193ms (compile: 104 ms, reload: 1693 ms, reassemble: 2183 ms).
F/libc    (12492): Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x7beb622080 in tid 12592 (1.ui), pid 12492 (amar.progress)
Process name is amar.progress, not key_process
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'OPPO/RMX1801/RMX1801:10/QKQ1.191014.001/1602573502:user/release-keys'
Revision: '0'
ABI: 'arm64'
Timestamp: 2023-11-02 10:20:55+0530
pid: 12492, tid: 12592, name: 1.ui  >>> amar.progress <<<
uid: 12708
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x7beb622080
    x0  0000007ad1799800  x1  0000000000200022  x2  0000000000000020  x3  000000000000000f
    x4  0000007b48f284a0  x5  0000007b48f284c0  x6  0000007ad1972d40  x7  0000007ad1972dc0
    x8  0000007beb622080  x9  ffffffffffffffff  x10 0000000000000001  x11 0000000000000000
    x12 0000000081957120  x13 0000007ad1972cc0  x14 0000007b48f27150  x15 0000007b48f27160
    x16 0000007afd587f10  x17 0000007be7f41cb0  x18 0000007ae9f96000  x19 0000007ad1799800
    x20 0000000000200022  x21 0000000000000002  x22 0000007ad19aa6c0  x23 0000007af0f57020
    x24 0000007af445e100  x25 0000007ad17cb5d0  x26 0000007ad17cb5d0  x27 0000007af0f555a8
    x28 0000007af0f555b8  x29 0000007af0f552c0
    sp  0000007af0f552b0  lr  0000007afd4ab4ec  pc  0000007afd532c58
backtrace:
      #00 pc 0000000000171c58  /data/app/amar.progress-5EAL6GEc1J-9Y1r6E1fQeQ==/lib/arm64/libobjectbox-jni.so (BuildId: e5f49ef68eb14db1)
      #01 pc 00000000000ea4e8  /data/app/amar.progress-5EAL6GEc1J-9Y1r6E1fQeQ==/lib/arm64/libobjectbox-jni.so (BuildId: e5f49ef68eb14db1)
      #02 pc 00000000000e2e40  /data/app/amar.progress-5EAL6GEc1J-9Y1r6E1fQeQ==/lib/arm64/libobjectbox-jni.so (BuildId: e5f49ef68eb14db1)
      #03 pc 00000000000e38ec  /data/app/amar.progress-5EAL6GEc1J-9Y1r6E1fQeQ==/lib/arm64/libobjectbox-jni.so (BuildId: e5f49ef68eb14db1)
      #04 pc 00000000000e5ef4  /data/app/amar.progress-5EAL6GEc1J-9Y1r6E1fQeQ==/lib/arm64/libobjectbox-jni.so (BuildId: e5f49ef68eb14db1)
      #05 pc 00000000000d6718  /data/app/amar.progress-5EAL6GEc1J-9Y1r6E1fQeQ==/lib/arm64/libobjectbox-jni.so (obx_store_close+100) (BuildId: e5f49ef68eb14db1)
      #06 pc 0000000001c06f4c  /data/app/amar.progress-5EAL6GEc1J-9Y1r6E1fQeQ==/lib/arm64/libflutter.so (BuildId: 59e80130e559b84528a7f4adadcde9e10efeac15)

Lost connection to device.

Exited.

[greenrobot-team]

@user97116 This does not look related. You also commented on other unrelated issues. Please create a new issue for your problem and share as much detail as possible.

[user97116]

@techouse Hi, when I am trying to open openStore it says read only can't open then I restart app then I got this issue, somehow I solved this issue