Hive Database Resets When Re-starting on Flutter Android Simulator

[mjacobsca]

(Almost) every time we hard stop and do a full restart of our app on the Android Simulator, our Hive DB gets deleted. I'd say it's more often than not, but it is inconsistent. Is this a bug?

[jessejay247]

Currently Happens on iOS too, its a sudden development, I think it may be linked with flutter and dart update. Did you update yours?

[mjacobsca]

Currently Happens on iOS too, its a sudden development, I think it may be linked with flutter and dart update. Did you update yours?

No. It happens randomly, without any DART/FLUTTER updates. One minute the db will be fine, and another restart a few minutes later all my user-prefs are gone because the .hive file no longer exists. I've tried stopping the app, cleaning project files, and the database is fine after these... sometimes. And then 20m later I do another restart and it's just gone. It's not consistent. I am at my wits end. It's making development frustrating. If I could be guaranteed that this was a simulator, development-only problem, I wouldn't worry as much. But I'm afraid when we start rolling out our app that this is going to happen to users.

[TimothyGCY]

I'm having likely the same issue but mine does not tie to time. All data lost whenever I kill the app and re-open it.

[nicoroy2561]

I am having the same issue in debug mode. Every time I restart the app and try to open the same box, it appears to be empty.

[mjacobsca]

I haven't noticed it happening in the production app. So maybe it is a simulator-only issue?

[nicoroy2561]

I should also specify that this happens on Flutter v2.0.6 with sound null safety active, when running on my debug device (not simulator). Edit: did not happen before sound null safety, possibly not even after updating Flutter (I'm not sure wether it happened on v2.0.2 but I did not notice it, so possibly not.)

[SaudQureshi1997]

facing the same issue in my android device (not simulated).

Edit: I did not notice it on dart sdk < 2.7.

[bokanian]

I have same issue and after restart the app and call the box name got: Box not found. Did you forget to call Hive.openBox()?

[jutarhon]

One of my customer complains every now and then that she has lost data, this happens on iOs device. Unfortunately I cannot get more information about the case since it is not giving any errors, it seems like app is just reset. But from code perspective it looks like that whole box is gone missing. IMO this is pretty critical error for database. Some sort of guess is that it might be related to starting app from e.g. push notification. Same customer also hitted -> https://github.com/hivedb/hive/issues/192 or https://github.com/hivedb/hive/issues/263 so this might be somehow related to this as well

[themisir]

One of my customer complains every now and then that she has lost data, this happens on iOs device. Unfortunately I cannot get more information about the case since it is not giving any errors, it seems like app is just reset. But from code perspective it looks like that whole box is gone missing. IMO this is pretty critical error for database. Some sort of guess is that it might be related to starting app from e.g. push notification. Same customer also hitted -> #192 or #263 so this might be somehow related to this as well

If you need high reliability I would suggest using:

• secure storage for storing credentials
• sqlite3 for storing structured data
• preferences for storing other simple key value data

Don't get me wrong but all these tools had been around lots of years (SQLite was around for about 21 years and secure storage package uses APIs baked into operation system which was used by lots of apps). And store some data on server side if possible. It's very hard to create reliable database with disaster recovery, speed and ease use. Hive is mostly good about ease of use and performance, but has issues with reliability.

[jutarhon]

I really like no-sql style approach here and overall Hive easiness compared to old school sql dbs. So lets see if I can give you some hints for stability. I just briefly checked through source, and seems like compact is done pretty much on end of writing etc. So it is first place for me to check if there is some possibilities for corruption. I checked only VM side of compact and there seems to be so that it writes compacted (or sorted) data to temporary file (which is good). But if I am right, writing happens in loop inside try catch, and even if there is any error (errors are silently ignored), it just renames temporary file as main file. Does this sound somewhat error prone to you? EDIT: Actually there is no catch, so it should not rename on error. Is there then possibility that it somehow writes empty file (without errors)? EDIT2: At least _compactionScheduled stays true after error, but that might not be problem EDIT3: Seems like there could be problems with File.exists call, as you create new file when file.exists returns false, this might clear the existing box... -> https://stackoverflow.com/questions/57655086/file-exists-returns-false-for-file-directory-that-actually-exists EDIT4: Maybe you could change those exists calls with simply trying to open file, if opening fails then create new file? EDIT5: Just assuming from the stackoverflow that exists can return false e.g. in case of there is no access for file. Maybe this can be case e.g. when launching app with push notification (and maybe it runs before user is e.g. opened screen lock). Just guessing, but I think that there might be quite a many cases where file.exists returns false... EDIT6: dart.io File: Future<bool> exists() { return _dispatchWithNamespace(_IOService.fileExists, [null, _rawPath]).then((response) { if (_isErrorResponse(response)) { throw _exceptionFromResponse(response, "Cannot check existence", path); } return response; }); } Looks like there is possibility to catch exception in case of some problems EDIT7: dart mac code seems to be using stat() for checking file existence (without retries) bool File::Exists(Namespace* namespc, const char* name) { struct stat st; if (NO_RETRY_EXPECTED(stat(name, &st)) == 0) { // Everything but a directory and a link is a file to Dart. return !S_ISDIR(st.st_mode) && !S_ISLNK(st.st_mode); } else { return false; } } and fstatat for android, here having temporary failure retry, but I think it does not help if it is returning e.g. access problem if (TEMP_FAILURE_RETRY(fstatat(ns.fd(), ns.path(), &st, 0)) == 0) { // Everything but a directory and a link is a file to Dart. return !S_ISDIR(st.st_mode) && !S_ISLNK(st.st_mode); } else { return false; }

[themisir]

Wow @jutarhon , thanks for such a great insights. I'll look at them. But it's really hard to work on this issue without having a minimum reproducible example. It happens randomly and that's the issue. We (at least I hadn't) couldn't produce this issue on our machines. So I lost my hope on this issue because it took so much time without any progress. But I'll try to look at it in a few days again.

[jutarhon]

I tried to reproduce customers problem, but as usual, it seems to be impossible... talked with customer and she said that problem occurs when app is opened via push message, and still quite rarely (of course). I also think about ios automatic app data backup, if there is some sort of locking for files, but this is just plain guessing. Most likely she does not have antivirus software, which seems to make some locking in some cases.

[shahzodjonweb]

When I have upgraded Dart SDK , app have caused the same issue. When I have downgraded the Dart SDK, Hive started working as expected. Solution is downgrading Dart SDK. My current SDK is:

[erayerdin]

When I have upgraded Dart SDK , app have caused the same issue. When I have downgraded the Dart SDK, Hive started working as expected. Solution is downgrading Dart SDK. My current SDK is:

This didn't work for me. Also see below:

https://github.com/hivedb/hive/issues/863#issuecomment-1008137593

[dreadjr]

We run into this in production and development as well. The data is on the filesystem.

[R98rayan]

this problem have it to me in my Windows project sometime by randomly the database reset and i don't know why, i have no idea how to solve it.

[ahmeedev]

also happens on my side, anyone have solution?

[rashmisridar]

Phasing this issue in the real devices too. Any solutions?

[ThiruDev50]

I am having the same issue in debug mode. Every time I restart the app and try to open the same box, it appears to be empty.

Did you got a solution to fix this?

[dragongesa]

if anyone still having this issue i have a workaround

[jamesdixon]

@dragongesa what is the workaround?

[dragongesa]

This is the only work version if you have auto_route, freezed and json_serializable together. if you still having this issue. please edit pubspect.lock by hand and forcing the version to match below. but it seems not work anymore in the latest flutter 3. im looking up for this again because i upgraded my flutter version.

#DEP hive: ^2.0.5 hive_flutter: ^1.0.0 freezed_annotation: json_annotation: #DEV auto_route_generator: ^2.1.0 build_runner: ^2.0.6 analyzer: freezed: ^0.14.1+3 hive_generator: ^1.1.0 json_serializable: ^4.1.0

[waqadArshad]

@jamesdixon did u try this workaround?

This is the only work version if you have auto_route, freezed and json_serializable together. if you still having this issue. please edit pubspect.lock by hand and forcing the version to match below. but it seems not work anymore in the latest flutter 3. im looking up for this again because i upgraded my flutter version.

#DEP hive: ^2.0.5 hive_flutter: ^1.0.0 freezed_annotation: json_annotation: #DEV auto_route_generator: ^2.1.0 build_runner: ^2.0.6 analyzer: freezed: ^0.14.1+3 hive_generator: ^1.1.0 json_serializable: ^4.1.0

[waqadArshad]

One of my customer complains every now and then that she has lost data, this happens on iOs device. Unfortunately I cannot get more information about the case since it is not giving any errors, it seems like app is just reset. But from code perspective it looks like that whole box is gone missing. IMO this is pretty critical error for database. Some sort of guess is that it might be related to starting app from e.g. push notification. Same customer also hitted -> #192 or #263 so this might be somehow related to this as well

If you need high reliability I would suggest using:

• secure storage for storing credentials
• sqlite3 for storing structured data
• preferences for storing other simple key value data

Don't get me wrong but all these tools had been around lots of years (SQLite was around for about 21 years and secure storage package uses APIs baked into operation system which was used by lots of apps). And store some data on server side if possible. It's very hard to create reliable database with disaster recovery, speed and ease use. Hive is mostly good about ease of use and performance, but has issues with reliability.

@themisir Don't get me wrong but it is really disappointing that you made this as a local database (for which high reliability is quite a very very very important feature), and now instead of owning up to it and try to fix the issue, you are recommending the developers to change their whole codebase that is built around this package.

[themisir]

@waqadArshad man, this is an open source project powered by many members of the community and neither of us owe anything to anyone. We all do our best to provide whatever we can without expecting anything in return. I started contributing to hive because I used it myself and found a value in using it and had spare time for contribution.

Regarding this issue, it's probably an edge case and without any way to reproduce it we have no way to find what's going on. There was similar case that a few months ago got fixed because someone on the community finally figured it out what's going on. Again, I can not provide any warranty that it's not going to fail as covered by the LICENSE.

you are recommending the developers to change their whole codebase that is built around this package

First I would highly encourage EVERYONE to not build their project around ANY package. It's just a door to disaster. If you're building something for hobby, sure do it. But if you have long term goals, wrap it around with your own layer so you can switch gears when needed.

Secondly I just provided recommendation because I thought it might be valuable. If you need reliability use something proven to be reliable (nothing beats time when it comes to proven reliability), if you wanna take risks sure do it. That DOESN'T means hive won't work, or would crash randomly. In my experience aside from #914 I didn't experienced any reliability issue with hive personally. However it's up to the developer to choose which way they would like to go. I can't just lie that it wouldn't ever fail. Even most proven reliable stuff from time to time gets broken, that's just nature of the software. (I spend multiple nights to learn importance of backups hard way, because our production postgres cluster got crashed and I had to manually recover it before our customers wake up. Yet nobody says "don't use postgres because it might fail").

Thirdly it's not an easy thing to build a reliable database. You're dealing with unknowns you're not even aware that might happen. Again if we see something wrong going on and if we have a way to reproduce it, we'll try to fixing it. If we can't we'll ask community for the help.

Anyways, I hope you understand the situation. I know the feeling you're going thru - it sucks when things randomly break. But complaining doesn't help anyone. If you have a solution you can open up a PR and contribute to the package you've built your whole project around.

EOF;

[waqadArshad]

@themisir really sorry about that, I just spent 3 to 4 days (of a tight deadline project) on fixing the bugs caused by this, without thinking that this could/would fail. and last night I realized that this might be the case and it is and that just made me really sad because, on top of those 3-4 days, I now need to put in at least 2 more to change it to sqflite and make sure that I change it everywhere and also make sure that it works.

I am really sorry to have said that. and I'll try and see if I can help with this issue as soon as I have some free time.

[themisir]

Hey @waqadArshad don't worry :) We've all been there. It's really stressful to work with tight deadlines. About the issue I'm sorry but I have no idea why it would reset by itself. Unless explicitly deleted the db should persist between restarts. I never experienced it myself, nor saw it happening on others who used hive.

Aside from that I would generally advise to always encapsulate storage system with a separate layer. It's really not a good idea to access data directly from places where you would need them.

[Sania-Developer]

When storing data it shows on the screen till I do not hot restart the app. Whenever I restart the app it shows an error. Did anyone solve this issue?

[LazyC0derX]

Facing similar issue,hive is not resetting but values inside model class is being changed to null. Issue is only with boxes built with custom model classes or adapter.I tried creating a box with string type and the values are persistent after hot restart.

Steps to reproduce:

I created a model

@Freezed()
@HiveType(typeId: 0)
class Bank with _$Bank {
  @JsonSerializable(includeIfNull: false, explicitToJson: true)
  const factory Bank({
    @HiveField(0) String? id,
    @HiveField(1) String? bankName,
    @HiveField(2) required bool? isVisible,
  }) = _Bank;

  factory Bank.fromJson(Map<String, dynamic> json) => _$BankFromJson(json);
}

then

bank=Bank(id:"1665301531499",name:"test",isVisible:true)

I tried box.put as await box.put(bank.id,bank)

and immedietly returned value using box.get and I got the value in ui.

but after restart the key is there "1665301531499" but value got changed into Bank(id:null,bankName:null,isVisible:null)

flutter version is 3.3.4

dependencies:
  auto_route: ^5.0.1
  cloud_firestore: ^3.5.1
  connectivity_plus: ^2.3.9
  cupertino_icons: ^1.0.5
  firebase_auth: ^3.11.2
  firebase_core: ^1.24.0
  flutter:
    sdk: flutter
  flutter_form_builder: ^7.7.0
  flutter_hooks: ^0.18.5+1
  flutter_svg: ^1.1.5
  form_builder_validators: ^8.3.0
  freezed_annotation: ^2.1.0
  get_it: ^7.2.0
  google_sign_in: ^5.4.2
  **hive_flutter: ^1.1.0**
  hooks_riverpod: ^2.0.2
  intl: ^0.17.0
  json_annotation: ^4.7.0
  path_provider: ^2.0.11
  shared_preferences: ^2.0.15


dev_dependencies:
  auto_route_generator: ^5.0.2
  build_runner: ^2.2.1
  flutter_lints: ^2.0.1
  flutter_test:
    sdk: flutter
  freezed: any
  **hive_generator: ^1.1.3**
  json_serializable: ^6.5.0