Migrate Hive v2 to support Flutter web WASM

[Rexios80]

trafficstars

Migrates Hive v2 to support Flutter web WASM since v3 and v4 are major changes (and not stable)

This should not be merged into the main branch, but there isn't a better option right now. This should probalby be merged into a branch created off of the tag v2.2.3 since that is the commit these changes are based on.

IF YOU WANT TO USE THIS NOW

I have released the following packages to replace Hive in my projects. The intent is to keep these up to date with modern Dart/Flutter standards.

hive_ce hive_ce_flutter hive_ce_generator

If you do not want to use Hive Community Edition, you can use this PR directly:

dependency_overrides:
  hive:
    git:
      url: https://github.com/Rexios80/hive
      ref: 4eb0dbc7807e7210831a2c71f056ac15c9e32e52
      path: hive
  hive_generator:
    git:
      url: https://github.com/Rexios80/hive
      ref: 4eb0dbc7807e7210831a2c71f056ac15c9e32e52
      path: hive_generator

Make sure to run the generator after updating

Related issues

https://github.com/isar/hive/issues/1287 https://github.com/isar/isar/issues/1617 https://github.com/isar/isar/pull/1616

[theniceboy]

Thank you for your effort! I've been trying to migrate Hive v2 to package:web and have been stuck. I'll be trying your branch very soon and giving you feedback.

[theniceboy]

I'm experience a runtime error. Here's the related info

Unsupported operation: Infinity or NaN toInt
main.dart.mjs:56     at Error._throwWithCurrentStackTrace (http://localhost:59048/main.dart.wasm:wasm-function[1606]:0x43ec60)
    at _BoxedDouble.toInt (http://localhost:59048/main.dart.wasm:wasm-function[1910]:0x4434a5)
    at JSNumberToNumber|get#toDartInt (http://localhost:59048/main.dart.wasm:wasm-function[1908]:0x443446)
    at StorageBackendJs.getKeys closure at file:///Users/david/.pub-cache/git/hive-456dab3bd156f7fa421aeb48884785ee2fa50471/hive/lib/src/backend/js/native/storage_backend_js.dart:126:45 (http://localhost:59048/main.dart.wasm:wasm-function[84040]:0xba5142)
    at closure wrapper at file:///Users/david/.pub-cache/git/hive-456dab3bd156f7fa421aeb48884785ee2fa50471/hive/lib/src/backend/js/native/storage_backend_js.dart:126:45 trampoline (http://localhost:59048/main.dart.wasm:wasm-function[84044]:0xba543b)
    at MappedListIterable.elementAt (http://localhost:59048/main.dart.wasm:wasm-function[2386]:0x44d35b)
    at ListIterator.moveNext (http://localhost:59048/main.dart.wasm:wasm-function[2380]:0x44d1eb)
    at new _GrowableList._ofEfficientLengthIterable (http://localhost:59048/main.dart.wasm:wasm-function[2234]:0x44a0a9)

Ran with flutter run -d chrome --debug --wasm using 3.23.0-13.0.pre.54

[Rexios80]

The --debug flag doesn't work right now. See https://github.com/flutter/flutter/issues/148850

But that's probably not the issue here. You re-ran the generator right?

[Rexios80]

That error seems to imply that the issue is with this line:

https://github.com/Rexios80/hive/blob/456dab3bd156f7fa421aeb48884785ee2fa50471/hive/lib/src/backend/js/native/storage_backend_js.dart#L128

But that would mean you somehow have a box key that's infinity or NaN which doesn't make any sense. I suppose we could just call toDartDouble there instead but that seems weird. Any thoughts on how the heck one of your keys is infinity or NaN?

[theniceboy]

I did forget to re-ran the generator. The error still shows up. The same code runs well when compiled to main.dart.js without error. Other than using the master channel, I'm not how can I get a useful stack trace.

Another exception I'm having is this:

Unsupported operation: Platform._operatingSystem
main.dart.mjs:56     at Error._throwWithCurrentStackTrace (http://localhost:58619/main.dart.wasm:wasm-function[1608]:0x43f39d)
    at _Platform._operatingSystem (http://localhost:58619/main.dart.wasm:wasm-function[4942]:0x47e16e)
    at _Platform.operatingSystem (http://localhost:58619/main.dart.wasm:wasm-function[4941]:0x47e160)
    at Platform.operatingSystem (http://localhost:58619/main.dart.wasm:wasm-function[4940]:0x47e153)
    at IOPlatform.operatingSystem (http://localhost:58619/main.dart.wasm:wasm-function[15067]:0x55706b)
    at Platform.isAndroid (http://localhost:58619/main.dart.wasm:wasm-function[15066]:0x557042)
    at _defaultSpotlightUrl (http://localhost:58619/main.dart.wasm:wasm-function[15065]:0x557026)
    at new Spotlight (initializer) (http://localhost:58619/main.dart.wasm:wasm-function[15063]:0x557011)

Which may or may not be related. I'm not sure which is the error that's preventing the app to run.

I made the change you mentioned, and now I'm getting this error:

JavaScriptError
main.dart.mjs:56     at StorageBackendJs.initialize inner (http://localhost:64486/main.dart.wasm:wasm-function[85445]:0xbd0841)
    at _awaitHelper closure at org-dartlang-sdk:///dart-sdk/lib/_internal/wasm/lib/async_patch.dart:83:16 (http://localhost:64486/main.dart.wasm:wasm-function[86161]:0xbe1a7b)
    at closure wrapper at org-dartlang-sdk:///dart-sdk/lib/_internal/wasm/lib/async_patch.dart:83:16 trampoline (http://localhost:64486/main.dart.wasm:wasm-function[86166]:0xbe1b3f)
    at _RootZone.runUnary (http://localhost:64486/main.dart.wasm:wasm-function[3417]:0x46e595)
    at _FutureListener.handleValue (http://localhost:64486/main.dart.wasm:wasm-function[3412]:0x46e388)
    at _Future._propagateToListeners closure handleValueCallback at org-dartlang-sdk:///dart-sdk/lib/async/future_impl.dart:859:33 (http://localhost:64486/main.dart.wasm:wasm-function[3390]:0x46dd29)
    at _Future._propagateToListeners (http://localhost:64486/main.dart.wasm:wasm-function[3387]:0x46d950)
    at _Future._completeWithValue (http://localhost:64486/main.dart.wasm:wasm-function[3442]:0x46ea5e)

Hope it helps a little

[Rexios80]

On Flutter stable you need to run flutter build web --wasm --no-strip-wasm or else the traces will be minified

[Rexios80]

The hive code has no call to Platform.isAndroid so that first exception is either an issue in your code or another dependency

I'm not sure what the second exception is about. Probably just another exception caused by the key that's infinity or NaN. Maybe try this:

/// Not part of public API
@visibleForTesting
Future<List<Object?>> getKeys({bool cursor = false}) async {
  var store = getStore(false);

  if (store.has('getAllKeys') && !cursor) {
    final result = await getStore(false).getAllKeys(null).asFuture();
    final keys = <Object?>[];
    for (final key in (result as JSArray).toDart) {
      if (key is JSNumber) {
        final keyDart = key.toDartDouble;
        if (keyDart.isFinite) {
          keys.add(keyDart.toInt());
        } else {
          print('Box key is not finite: $key');
        }
      } else if (key is JSString) {
        keys.add(key.toDart);
      }
    }
    return keys;
  } else {
    final cursors = await store.getCursors();
    return cursors.map((e) => e.key).toList();
  }
}

What I'm afraid of is that you will be missing data due to keys not getting read correctly or something. Let me know how many times that print statement gets called.

[theniceboy]

Thanks! I tried the flutter build web --wasm --no-strip-wasm command on stable. The Platform._operatingSystem exception is gone, and the Infinity or NaN toInt exception is also gone. Now the only exception I got was this:

JavaScriptError
main.dart.mjs:56     at FirebaseCoreWeb.initializeApp inner (http://localhost:5066/main.dart.wasm:wasm-function[43670]:0x7d6183)
    at _awaitHelperWithTypeCheck closure at org-dartlang-sdk:///dart-sdk/lib/_internal/wasm/lib/async_patch.dart:97:16 (http://localhost:5066/main.dart.wasm:wasm-function[2425]:0x37ecac)
    at closure wrapper at org-dartlang-sdk:///dart-sdk/lib/_internal/wasm/lib/async_patch.dart:97:16 trampoline (http://localhost:5066/main.dart.wasm:wasm-function[2431]:0x37edad)
    at _RootZone.runUnary (http://localhost:5066/main.dart.wasm:wasm-function[2467]:0x37f8f3)
    at _Future._propagateToListeners (http://localhost:5066/main.dart.wasm:wasm-function[2448]:0x37f4e1)
    at _Future._completeWithValue (http://localhost:5066/main.dart.wasm:wasm-function[2482]:0x37fb93)
    at _Future._asyncCompleteWithValue closure at org-dartlang-sdk:///dart-sdk/lib/async/future_impl.dart:721:29 (http://localhost:5066/main.dart.wasm:wasm-function[2809]:0x384e68)
    at closure wrapper at org-dartlang-sdk:///dart-sdk/lib/async/future_impl.dart:721:29 trampoline (http://localhost:5066/main.dart.wasm:wasm-function[2811]:0x384e7f)

Seemed like there's a bug in the Dart compiler: https://github.com/firebase/flutterfire/issues/12623

It's fixed in the master channel, however I get this error again when I switch to the master channel:

Unsupported operation: Platform._operatingSystem
main.dart.mjs:56     at Error._throwWithCurrentStackTrace (http://localhost:5066/main.dart.wasm:wasm-function[1420]:0x2f5162)
    at Platform.operatingSystem (http://localhost:5066/main.dart.wasm:wasm-function[3200]:0x31f85a)
    at Platform.isWindows (http://localhost:5066/main.dart.wasm:wasm-function[4264]:0x347016)
    at Spotlight (http://localhost:5066/main.dart.wasm:wasm-function[8152]:0x3abef6)
    at SentryFlutter.init inner (http://localhost:5066/main.dart.wasm:wasm-function[43777]:0x7680b7)
    at main inner.1 (http://localhost:5066/main.dart.wasm:wasm-function[9200]:0x3cce3b)
    at _awaitHelperWithTypeCheck closure at org-dartlang-sdk:///dart-sdk/lib/_internal/wasm/lib/async_patch.dart:97:16 (http://localhost:5066/main.dart.wasm:wasm-function[2428]:0x31003e)
    at closure wrapper at org-dartlang-sdk:///dart-sdk/lib/_internal/wasm/lib/async_patch.dart:97:16 trampoline (http://localhost:5066/main.dart.wasm:wasm-function[2434]:0x31013f)

This time it's sentry. I don't think either is related to Hive, but at the moment I'm unable to test out this branch with my app unfortunately.

[theniceboy]

Filed this issue: https://github.com/getsentry/sentry-dart/issues/2082

(Not that it's related to Hive. Just for tracking sake)

[Rexios80]

Can you take out sentry for now and see if the app runs?

Also did you have to edit the getKeys method?

[theniceboy]

Yes, I updated the getKeys method with the code you provided.

I got rid of Sentry. Now this is the error I'm getting:

JavaScriptError
main.dart.mjs:56     at StorageBackendJs.initialize inner (http://localhost:5066/main.dart.wasm:wasm-function[42752]:0x73513c)
    at _awaitHelperWithTypeCheck closure at org-dartlang-sdk:///dart-sdk/lib/_internal/wasm/lib/async_patch.dart:97:16 (http://localhost:5066/main.dart.wasm:wasm-function[2415]:0x300643)
    at closure wrapper at org-dartlang-sdk:///dart-sdk/lib/_internal/wasm/lib/async_patch.dart:97:16 trampoline (http://localhost:5066/main.dart.wasm:wasm-function[2421]:0x300736)
    at _RootZone.runUnary (http://localhost:5066/main.dart.wasm:wasm-function[2457]:0x301294)
    at _Future._propagateToListeners (http://localhost:5066/main.dart.wasm:wasm-function[2438]:0x300e66)
    at _Future._completeWithValue (http://localhost:5066/main.dart.wasm:wasm-function[2472]:0x301533)
    at _Future._asyncCompleteWithValue closure at org-dartlang-sdk:///dart-sdk/lib/async/future_impl.dart:735:29 (http://localhost:5066/main.dart.wasm:wasm-function[2784]:0x30642d)
    at closure wrapper at org-dartlang-sdk:///dart-sdk/lib/async/future_impl.dart:735:29 trampoline (http://localhost:5066/main.dart.wasm:wasm-function[2786]:0x306444)

[Rexios80]

Can you try/catch your openBox calls and print the error/stacktrace?

[theniceboy]

They're already in a try-catch block. Are you sure the error's from the openBox calls?

[Rexios80]

That should be what's calling backend.initialize but I'm not sure

[Rexios80]

Your code doesn't happen to be open source does it? Or can you make a minimal reproducible sample?

[theniceboy]

I'll get to that tomorrow. Does this fork work for you?

[Rexios80]

This branch works for me (at least I haven't noticed any issues) and all the tests pass (except a couple that test behavior for really old browsers) so I'm not sure why you're having issues

[theniceboy]

I see. Thanks for the help though!

[1l0]

Thanks for your effort. I have encountered the following error on running a built web app with build web --wasm.

main.dart.js:4953 Uncaught 
    at Object.e (https://localhost:4430/main.dart.js:2956:19)
    at Object.aa (https://localhost:4430/main.dart.js:2965:15)
    at https://localhost:4430/main.dart.js:48134:3
    at XM.a (https://localhost:4430/main.dart.js:4259:63)
    at XM.$2 (https://localhost:4430/main.dart.js:25287:14)
    at Object.M (https://localhost:4430/main.dart.js:4245:10)
    at v_.T9 (https://localhost:4430/main.dart.js:48140:10)
    at v_.u8 (https://localhost:4430/main.dart.js:48120:21)
    at https://localhost:4430/main.dart.js:16602:14
    at XM.a (https://localhost:4430/main.dart.js:4259:63)

Reproducible code here: https://github.com/1l0/hive_testbed

[Rexios80]

Please build with --no-strip-wasm to get a meaningful stack trace

[Rexios80]

Also try running with flutter master and the command flutter run -d chrome --wasm

Although your issue is probably due to using BoxCollection which I'm not sure is covered by tests

[Rexios80]

@1l0 @theniceboy

Please try commit hash 888f810f5be0f3f25ec9d1c90cf113214b801d82. I think the issue was with doing type checking incorrectly.

[Rexios80]

Unfortunately this does require a Dart version constraint of ^3.4.0, but I'm not sure we should care all that much since you need modern dart to compile to WASM anyways

[Rexios80]

I figured out how to run the tests compiled to WASM and fixed even more issues. Please let me know how testing goes.

[Rexios80]

@1l0 I wrote tests for BoxCollection and they pass so the original code you showed me should work now

[Rexios80]

I did a bunch of cleanup. I also refactored the code that required dart 3.4.0, but the web package requires dart 3.3.0 anyways so not really much of an accomplishment.

[1l0]

I have confirmed that it just works ™️. Errors are gone. I don't know why but with build web --wasm Chrome would chose js as a fallback instead of wasm though. https://github.com/flutter/flutter/issues/142822 might be related.

[theniceboy]

Just tried, it worked!

[Rexios80]

@1l0 https://docs.flutter.dev/platform-integration/web/wasm#serve-the-output-with-an-http-server

[1l0]

@1l0 https://docs.flutter.dev/platform-integration/web/wasm#serve-the-output-with-an-http-server

Thanks, I overlooked that information. The problem seems to be resolved.