LiteDB [BUG] System.Exception: LiteDB ENSURE: empty page must be defined as empty type

[BUG] System.Exception: LiteDB ENSURE: empty page must be defined as empty type

Open GW-FUB opened this issue 3 years ago • 40 comments

Version LiteDB v5.0.10 .NET Standard 2.1 Xamarin.Forms v5.0.0.2012

Describe the bug The database seems to get corrupted when I insert data. Here are multiple examples:


System.Exception: LiteDB ENSURE: empty page must be defined as empty type
  at LiteDB.Constants.ENSURE (System.Boolean conditional, System.String message) [0x0002a] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.Snapshot.NewPage[T] () [0x000a3] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.Snapshot.GetFreeDataPage (System.Int32 bytesLength) [0x00081] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.DataService+<>c__DisplayClass3_0+<<Insert>g__source|0>d.MoveNext () [0x00052] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.BufferWriter..ctor (System.Collections.Generic.IEnumerable`1[T] source) [0x00029] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.DataService.Insert (LiteDB.BsonDocument doc) [0x0005d] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine.InsertDocument (LiteDB.Engine.Snapshot snapshot, LiteDB.BsonDocument doc, LiteDB.BsonAutoId autoId, LiteDB.Engine.IndexService indexer, LiteDB.Engine.DataService data) [0x00094] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine+<>c__DisplayClass7_0.<Insert>b__0 (LiteDB.Engine.TransactionService transaction) [0x00076] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine.AutoTransaction[T] (System.Func`2[T,TResult] fn) [0x00055] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine.Insert (System.String collection, System.Collections.Generic.IEnumerable`1[T] docs, LiteDB.BsonAutoId autoId) [0x00055] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.LiteCollection`1[T].Insert (System.Collections.Generic.IEnumerable`1[T] entities) [0x00027] in <8e1f39e9bcf3452cab277acc385e2e12>:0 


System.Exception: LiteDB ENSURE: empty page must be defined as empty type
  at LiteDB.Constants.ENSURE (System.Boolean conditional, System.String message) [0x0002a] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.Snapshot.NewPage[T] () [0x000a3] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.Snapshot.GetFreeDataPage (System.Int32 bytesLength) [0x00081] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.DataService+<>c__DisplayClass3_0+<<Insert>g__source|0>d.MoveNext () [0x00052] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.BufferWriter..ctor (System.Collections.Generic.IEnumerable`1[T] source) [0x00029] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.DataService.Insert (LiteDB.BsonDocument doc) [0x0005d] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine.InsertDocument (LiteDB.Engine.Snapshot snapshot, LiteDB.BsonDocument doc, LiteDB.BsonAutoId autoId, LiteDB.Engine.IndexService indexer, LiteDB.Engine.DataService data) [0x00094] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine+<>c__DisplayClass7_0.<Insert>b__0 (LiteDB.Engine.TransactionService transaction) [0x00076] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine.AutoTransaction[T] (System.Func`2[T,TResult] fn) [0x00055] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine.Insert (System.String collection, System.Collections.Generic.IEnumerable`1[T] docs, LiteDB.BsonAutoId autoId) [0x00055] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.LiteCollection`1[T].Insert (System.Collections.Generic.IEnumerable`1[T] entities) [0x00027] in <8e1f39e9bcf3452cab277acc385e2e12>:0 


System.Exception: LiteDB ENSURE: empty page must be defined as empty type
  at LiteDB.Constants.ENSURE (System.Boolean conditional, System.String message) [0x0002a] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.Snapshot.NewPage[T] () [0x000a3] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.Snapshot.GetFreeDataPage (System.Int32 bytesLength) [0x00081] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.DataService+<>c__DisplayClass3_0+<<Insert>g__source|0>d.MoveNext () [0x00052] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.BufferWriter..ctor (System.Collections.Generic.IEnumerable`1[T] source) [0x00029] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.DataService.Insert (LiteDB.BsonDocument doc) [0x0005d] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine.InsertDocument (LiteDB.Engine.Snapshot snapshot, LiteDB.BsonDocument doc, LiteDB.BsonAutoId autoId, LiteDB.Engine.IndexService indexer, LiteDB.Engine.DataService data) [0x00094] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine+<>c__DisplayClass7_0.<Insert>b__0 (LiteDB.Engine.TransactionService transaction) [0x00076] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine.AutoTransaction[T] (System.Func`2[T,TResult] fn) [0x00055] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine.Insert (System.String collection, System.Collections.Generic.IEnumerable`1[T] docs, LiteDB.BsonAutoId autoId) [0x00055] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.LiteCollection`1[T].Insert (T entity) [0x0002e] in <8e1f39e9bcf3452cab277acc385e2e12>:0 


System.AggregateException: One or more errors occurred. (LiteDB ENSURE: empty page must be defined as empty type) ---> System.Exception: LiteDB ENSURE: empty page must be defined as empty type
  at LiteDB.Constants.ENSURE (System.Boolean conditional, System.String message) [0x0002a] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.Snapshot.NewPage[T] () [0x000a3] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.Snapshot.GetFreeDataPage (System.Int32 bytesLength) [0x00081] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.DataService+<>c__DisplayClass3_0+<<Insert>g__source|0>d.MoveNext () [0x00052] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.BufferWriter..ctor (System.Collections.Generic.IEnumerable`1[T] source) [0x00029] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.DataService.Insert (LiteDB.BsonDocument doc) [0x0005d] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine.InsertDocument (LiteDB.Engine.Snapshot snapshot, LiteDB.BsonDocument doc, LiteDB.BsonAutoId autoId, LiteDB.Engine.IndexService indexer, LiteDB.Engine.DataService data) [0x00094] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine+<>c__DisplayClass7_0.<Insert>b__0 (LiteDB.Engine.TransactionService transaction) [0x00076] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine.AutoTransaction[T] (System.Func`2[T,TResult] fn) [0x00055] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine.Insert (System.String collection, System.Collections.Generic.IEnumerable`1[T] docs, LiteDB.BsonAutoId autoId) [0x00055] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.LiteCollection`1[T].Insert (T entity) [0x0002e] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.Exception: LiteDB ENSURE: empty page must be defined as empty type
  at LiteDB.Constants.ENSURE (System.Boolean conditional, System.String message) [0x0002a] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.Snapshot.NewPage[T] () [0x000a3] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.Snapshot.GetFreeDataPage (System.Int32 bytesLength) [0x00081] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.DataService+<>c__DisplayClass3_0+<<Insert>g__source|0>d.MoveNext () [0x00052] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.BufferWriter..ctor (System.Collections.Generic.IEnumerable`1[T] source) [0x00029] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.DataService.Insert (LiteDB.BsonDocument doc) [0x0005d] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine.InsertDocument (LiteDB.Engine.Snapshot snapshot, LiteDB.BsonDocument doc, LiteDB.BsonAutoId autoId, LiteDB.Engine.IndexService indexer, LiteDB.Engine.DataService data) [0x00094] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine+<>c__DisplayClass7_0.<Insert>b__0 (LiteDB.Engine.TransactionService transaction) [0x00076] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine.AutoTransaction[T] (System.Func`2[T,TResult] fn) [0x00055] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.Engine.LiteEngine.Insert (System.String collection, System.Collections.Generic.IEnumerable`1[T] docs, LiteDB.BsonAutoId autoId) [0x00055] in <8e1f39e9bcf3452cab277acc385e2e12>:0 
  at LiteDB.LiteCollection`1[T].Insert (T entity) [0x0002e] in <8e1f39e9bcf3452cab277acc385e2e12>:0

Code to Reproduce Unfortunately I cannot share code, since I could not reproduce it easily in a test project and I cannot share my business work. I can just say that I'm pretty sure I'm not doing any kind of magic with LiteDB; I'm using it basically just to store data (delete all, re insert, get all)

Expected behavior No exception.

Feb 17 '21 10:02 GW-FUB

I'm getting this too, also the same workflow - delete and a bulk re-insert

Feb 18 '21 13:02 nabeelio

I'm getting a ton of ENSURE exceptions. If I start the app first, everything works fine. I guess, because there is not data yet and with my login I'm loading and using this data. Then after a restart the madness begins with ENSURE exceptions.

Also got this beauty now:

System.Exception: LiteDB ENSURE: if there is no items, HighestIndex must be clear
  at LiteDB.Constants.ENSURE (System.Boolean conditional, System.String message) [0x00015] in <e37eae51170e472a88b7e61cb2c985c3>:0 
  at LiteDB.Engine.BasePage.Delete (System.Byte index) [0x0010b] in <e37eae51170e472a88b7e61cb2c985c3>:0 
  at LiteDB.Engine.DataPage.DeleteBlock (System.Byte index) [0x00000] in <e37eae51170e472a88b7e61cb2c985c3>:0 
  at LiteDB.Engine.DataService.Delete (LiteDB.Engine.PageAddress blockAddress) [0x00020] in <e37eae51170e472a88b7e61cb2c985c3>:0 
  at LiteDB.Engine.LiteEngine+<>c__DisplayClass3_0.<Delete>b__0 (LiteDB.Engine.TransactionService transaction) [0x00078] in <e37eae51170e472a88b7e61cb2c985c3>:0 
  at LiteDB.Engine.LiteEngine.AutoTransaction[T] (System.Func`2[T,TResult] fn) [0x00037] in <e37eae51170e472a88b7e61cb2c985c3>:0 
  at LiteDB.Engine.LiteEngine.Delete (System.String collection, System.Collections.Generic.IEnumerable`1[T] ids) [0x00046] in <e37eae51170e472a88b7e61cb2c985c3>:0 
  at LiteDB.Engine.LiteEngine.DeleteMany (System.String collection, LiteDB.BsonExpression predicate) [0x000d9] in <e37eae51170e472a88b7e61cb2c985c3>:0 
  at LiteDB.LiteCollection`1[T].DeleteAll () [0x00000] in <e37eae51170e472a88b7e61cb2c985c3>:0

Feb 22 '21 15:02 GW-FUB

Adding a stacktrace too, I've got a ton of these. Looks like most of the time it's happening when there isn't a database/file yet.

LiteDB ENSURE: page type must be collection page in Constants.ENSURE (Boolean conditional, String message)

LiteDB.Constants.ENSURE(Boolean conditional, String message);LiteDB
LiteDB.Engine.CollectionPage..ctor(PageBuffer buffer);LiteDB.Engine.CollectionPage
LiteDB.Engine.BasePage.ReadPage[T](PageBuffer buffer);LiteDB.Engine
LiteDB.Engine.Snapshot.ReadPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion);LiteDB.Engine
LiteDB.Engine.Snapshot.GetPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion);LiteDB.Engine
LiteDB.Engine.CollectionService.Get(String name, Boolean addIfNotExists, CollectionPage& collectionPage);LiteDB.Engine
LiteDB.Engine.Snapshot..ctor(LockMode mode, String collectionName, HeaderPage header, UInt32 transactionID, TransactionPages transPages, LockService locker, WalIndexService walIndex, DiskReader reader, Boolean addIfNotExists);LiteDB.Engine.Snapshot
LiteDB.Engine.TransactionService.g__create|42_0(<>c__DisplayClass42_0& );LiteDB.Engine
LiteDB.Engine.TransactionService.CreateSnapshot(LockMode mode, String collection, Boolean addIfNotExists);LiteDB.Engine
LiteDB.Engine.LiteEngine.<>c__DisplayClass5_0.b__0(TransactionService transaction);LiteDB.Engine.LiteEngine
LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn);LiteDB.Engine
LiteDB.LiteCollection`1.EnsureIndex(String name, BsonExpression expression, Boolean unique);LiteDB
Acars.AirportDatabase.AirportDatabase.Save();Acars.AirportDatabase
Acars.AirportDatabase.AirportDatabase.CreateAndGenerateDatabase(SimType type

Feb 22 '21 16:02 nabeelio

Hi guys, this ensure error avoid/detect corruption state. I will review this operations to try reproduce this error.

Feb 22 '21 18:02 mbdavid

Thanks @mbdavid. This is happening on a brand new/fresh DB, or when deleting.

Another type:

System.Exception: LiteDB ENSURE: transaction must be active to rollback (current state: Disposed) in Constants.ENSURE (Boolean conditional, String message)
LiteDB.Constants.ENSURE(Boolean conditional, String message);LiteDB
LiteDB.Engine.TransactionService.Rollback();LiteDB.Engine
LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn);LiteDB.Engine
LiteDB.LiteCollection`1.Delete(BsonValue id);LiteDB
Acars.Database.DatabaseInterface.Delete[T](String tableName, Int32 id);Acars.Database
Acars.Services.PirepService.Delete(Pirep pirep);Acars.Services

Where this is the delete method Acars.Database.DatabaseInterface.Delete[T](String tableName, Int32 id);. And _lock is static.

        public void Delete<T>(string tableName, int id) where T : Model
        {
            lock (_lock) {
                var collection = GetCollection<T>(tableName);
                collection.Delete(id);
            }
        }

Feb 22 '21 18:02 nabeelio

I think that these errors occur when you first make a DeleteAll and afterwards an Insert (I usually use a Replace extensions method for this). But they first occur after a complete app restart. Before that everything works fine at first app launch.

Feb 23 '21 06:02 GW-FUB

Another thing I noticed: I'm not totally sure what exactly is the thing.

Usually my replace method looks like this:

collection.DeleteAll();
collection.Insert(newItems);

I got the feeling that the DeleteAll method got some problems with writing correctly. OR the problem is that deleting and inserting happens to fast 🤔

I implemented the method to first get all existing items from the database, compare to the new ones, find Inserts, Updates and Deletes and then change only what has to be changed. And this seems to work now 👍

Feb 23 '21 08:02 GW-FUB

If you are interested in my method you can see it here: https://gist.github.com/GW-FUB/d7cfa866023e82f6538bbf83047c331e

Feb 23 '21 08:02 GW-FUB

Hi @GW-FUB. maybe this DeleteAll has some problem. I tried to replicate using many different types of insert/delete all but don't get this error. Will be great if you tell me:

How many collections have your database?
Which is avg of your document size?
Do you use Singleton database instance? Or you open/close after each operation?
Do you have more indexes in this collection?
Do you change any pragma parameter?
How many documents you have in this insert/delete operation?

I will keep trying reproduce this error along this week

Feb 24 '21 21:02 mbdavid

Hey @mbdavid ,

of course I will tell you what I can. I got this error on different collections and at the moment I have fixed it, but I will try to gather as much information as possible :-)

How many collections have your database?

Around 23. I'm not totally sure as I got some default ones in my library that I'm not using in every app. I have split my database in three single ones:

UserData
UserFiles
AppData

Each one of them has its own file. That makes it easier to reset if something goes wrong (or e.g. to make a backup of user data without the files).

Which is avg of your document size?

Around 250 - 500 bytes. And there are usually 20 - 50 of them. So not a lot.

Do you use Singleton database instance? Or you open/close after each operation?

Yes I use a singleton. I added this in my Gist example.

Do you have more indexes in this collection?

Usually I use 0 to 2 indices in one collection. Also added an example.

Do you change any pragma parameter?

No, I don't use any pragmas. Not sure if UtcDate (database property) also counts.

How many documents you have in this insert/delete operation?

As I said: Usually 20 - 50

I'm really confused for this thing as I don't think I'm doing anything special and also not using a lot of data. Also in my other apps I didn't have any problems with v5, just in this one; I really hope I'm not missing anything important 😅

Feb 25 '21 07:02 GW-FUB

If you want I can send you the database classes, because they're quite isolated and there shouldn't be any "business" code

Feb 25 '21 07:02 GW-FUB

Hi @GW-FUB, thanks for the infos. I'm sure that you are not doing wrong... because ENSURE tests was added in 5.0.10 to avoid/test database corruption. So, if any exception like this occurs, I must take a look better.

Using your use-context I try reporduce this error in a long loop-sequence of operations. I will back as sson as I reproduce this. Thanks in advice

Feb 25 '21 11:02 mbdavid

Just also adding in case it helps - I'm using a single database file for storing several different collections. It's happening in a separate thread, using InsertBulk. I'm deleting the file first instead of using DeleteAll - not sure if that's a problem, but it was faster. But here is that code, and the corresponding exception:

        public void Save()
        {
            var stopwatch = new Stopwatch();
            Logger.Info("Saving airport database");

            lock (_lock) {

                stopwatch.Start();

                var dbFile = GetDbName(SimType);

                // Delete the old db file. Don't bother with updating rows
                if(File.Exists(dbFile)) {
                    File.Delete(dbFile);
                }

                //var connStr = $"Filename={dbFile};InitialSize=250MB";
                var connStr = dbFile;
                Logger.Debug($"Connection string: {connStr}");

                using (var db = new LiteDatabase(connStr)) {
                    var airports = db.GetCollection<FsAirport>("airports");
                    var runways = db.GetCollection<FsRunway>("runways");
                    var gates = db.GetCollection<FsGate>("gates");

                    airports.EnsureIndex("ICAO");
                    runways.EnsureIndex("ICAO");
                    gates.EnsureIndex("ICAO");

                    airports.InsertBulk(Airports);
                    foreach(var apt in Airports) {
                        if (apt.Runways.Count > 0) {
                            runways.InsertBulk(apt.Runways);
                        }

                        if (apt.Gates.Count > 0) {
                            gates.InsertBulk(apt.Gates);
                        }
                    }
                }

                stopwatch.Stop();
                Logger.Info($"Saving airport data time {stopwatch.ElapsedMilliseconds}ms");
            }
        }

And the exception (just reposting it from above)

LiteDB ENSURE: page type must be collection page in Constants.ENSURE (Boolean conditional, String message)

LiteDB.Constants.ENSURE(Boolean conditional, String message);LiteDB
LiteDB.Engine.CollectionPage..ctor(PageBuffer buffer);LiteDB.Engine.CollectionPage
LiteDB.Engine.BasePage.ReadPage[T](PageBuffer buffer);LiteDB.Engine
LiteDB.Engine.Snapshot.ReadPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion);LiteDB.Engine
LiteDB.Engine.Snapshot.GetPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion);LiteDB.Engine
LiteDB.Engine.CollectionService.Get(String name, Boolean addIfNotExists, CollectionPage& collectionPage);LiteDB.Engine
LiteDB.Engine.Snapshot..ctor(LockMode mode, String collectionName, HeaderPage header, UInt32 transactionID, TransactionPages transPages, LockService locker, WalIndexService walIndex, DiskReader reader, Boolean addIfNotExists);LiteDB.Engine.Snapshot
LiteDB.Engine.TransactionService.g__create|42_0(<>c__DisplayClass42_0& );LiteDB.Engine
LiteDB.Engine.TransactionService.CreateSnapshot(LockMode mode, String collection, Boolean addIfNotExists);LiteDB.Engine
LiteDB.Engine.LiteEngine.<>c__DisplayClass5_0.b__0(TransactionService transaction);LiteDB.Engine.LiteEngine
LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn);LiteDB.Engine
LiteDB.LiteCollection`1.EnsureIndex(String name, BsonExpression expression, Boolean unique);LiteDB
Acars.AirportDatabase.AirportDatabase.Save();Acars.AirportDatabase
Acars.AirportDatabase.AirportDatabase.CreateAndGenerateDatabase(SimType type

In this case - the airports that are saved are about 30-35k records, the runways and gates are probably 50-60k records. It ends up being really slow - I have an SSD and it takes a few minutes sometimes (the file, in the end, is about 200MB) But that's fine. You can see it's failing in trying to create the index.

Has this changed from 4.x? I haven't changed this code from then.

Feb 25 '21 12:02 nabeelio

Hi @nabeelio, I will try your example too. In you case seams in EnsureIndex method.

DeleteAll is slower way to delete all records from a collection, but it's only way to use inside a transaction. If you don't need run another operations in same transaction, you can use DropCollection or even delete you phisical file (fastest way but drop all database).

v5 works different from v4 because use better memory cache, WAL (write ahead log) and sql syntax... but API changes are minimal (and you got any compilation error).

Feb 25 '21 13:02 mbdavid

@mbdavid Thanks, I posted the other case where it's happening above in a Delete() method.

Yeah I'm just deleting the database file since it's the fastest. It's basically a write-once operation and then only reads after that, until the user needs to reload the data from the source files, which doesn't happen often. But from what I've seen, this has been on databases that aren't there.

Feb 25 '21 13:02 nabeelio

@nabeelio Have you tried deleting the log file when you delete the data file?

Feb 25 '21 14:02 lbnascimento

@lbnascimento I don't see a log file with those database files in that directory. It's the _nav files. What's weird is that settings.db does have one. The Delete() error I posted happens with the settings.db file. The other _nav files

Feb 25 '21 19:02 nabeelio

Hi @nabeelio, the log file are deleted when database are closed (when LiteDB can execute checkpoint operation). Checkpoint command override datafile with log file. After this command, log file are deleted (or set to 0).

If you can test, try use 5.1 branch version. This new version removes log file (log data are inserted inside same data file).

Mar 01 '21 20:03 mbdavid

I will see if I can consistently reproduce it; it's a customer that is running into it and I only get the crash reports.

Mar 02 '21 00:03 nabeelio

Hello,

I'm getting "LiteDB ENSURE : empty page must be defined as empty type" exception too. It s occuring in a database, where I keep inserting, uploading and deleting documents. I'm in v5.0.10.

System.Exception: LiteDB ENSURE: empty page must be defined as empty type
   at LiteDB.Constants.ENSURE(Boolean conditional, String message)
   at LiteDB.Engine.Snapshot.NewPage[T]()
   at LiteDB.Engine.Snapshot.GetFreeDataPage(Int32 bytesLength)
   at LiteDB.Engine.DataService.<>c__DisplayClass3_0.<<Insert>g__source|0>d.MoveNext()
   at LiteDB.Engine.BufferWriter..ctor(IEnumerable`1 source)
   at LiteDB.Engine.DataService.Insert(BsonDocument doc)
   at LiteDB.Engine.LiteEngine.InsertDocument(Snapshot snapshot, BsonDocument doc, BsonAutoId autoId, IndexService indexer, DataService data)
   at LiteDB.Engine.LiteEngine.<>c__DisplayClass7_0.<Insert>b__0(TransactionService transaction)
   at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn)
   at LiteDB.LiteCollection`1.Insert(T entity)
   at LiteDB.LiteFileStream`1.WriteChunks(Boolean flush)
   at LiteDB.LiteFileStream`1.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at LiteDB.LiteStorage`1.Upload(TFileId id, String filename, Stream stream, BsonDocument metadata)

Mar 19 '21 08:03 alb3ric

I'm getting this issue when using Library with Hangfire.LiteDb. It seems that some entries end-up on the wrong page since the row 2 is the server status entry, not lock table entry

Apr 03 '21 15:04 temnava

Any news about this issue? I'm getting this error message:

LiteDB ENSURE: transaction must be active to rollback (current state: Disposed)

when executing EnsureIndex on v5.0.10

System.Exception: LiteDB ENSURE: transaction must be active to rollback (current state: Disposed) at LiteDB.Constants.ENSURE(Boolean conditional, String message) at LiteDB.Engine.TransactionService.Rollback() at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func2 fn) at LiteDB.Engine.LiteEngine.EnsureIndex(String collection, String name, BsonExpression expression, Boolean unique) at LiteDB.LiteCollection1.EnsureIndex(String name, BsonExpression expression, Boolean unique) at LiteDB.LiteCollection1.EnsureIndex(BsonExpression expression, Boolean unique) at Relationship.Promotion.Applier.Infrastructure.Databases.LiteContext1.GetCollection() in /home/vsts/work/1/s/RelationshipPromotionApplier/src/Relationship.Promotion.Applier.Infrastructure/Databases/LiteContext.cs:line 41

Jul 12 '21 22:07 rasert

This is happening too...

System.Threading.SynchronizationLockException HResult=0x80131518 Message=Object synchronization method was called from an unsynchronized block of code. Source=System.Private.CoreLib StackTrace: at System.Threading.Monitor.Exit(Object obj) at LiteDB.Engine.LockService.ExitLock(String collectionName) at LiteDB.Engine.Snapshot.Dispose() at LiteDB.Engine.TransactionService.Rollback() at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func2 fn) at LiteDB.Engine.LiteEngine.EnsureIndex(String collection, String name, BsonExpression expression, Boolean unique) at LiteDB.LiteCollection1.EnsureIndex(String name, BsonExpression expression, Boolean unique) at LiteDB.LiteCollection1.EnsureIndex(BsonExpression expression, Boolean unique) at Relationship.Promotion.Applier.Infrastructure.Databases.LiteContext1.GetCollection() in C:\Users\jmourao\workspace\MotorPromocao\RelationshipPromotionApplier\src\Relationship.Promotion.Applier.Infrastructure\Databases\LiteContext.cs:line 56

Jul 13 '21 20:07 rasert

@rasert Could you test with the latest version?

Jul 21 '21 19:07 lbnascimento

Hello,

Any update to this one? I got very similar issue LiteDB ENSURE: page type **must be index page** on 5.0.8 database. I got this error message from LiteDb Studio v. 1.0.2 when debugging the database table.

I got a little different error message when rebuilding the same database: LiteDB.LiteException: Invalid Data on 74699. Full zero: False. Page Type: Empty. Prev/Next: 4294967295/74700. UniqueID: 0. ShareCounter: 0. at LiteDB.Engine.DataPage..ctor(PageBuffer buffer) at LiteDB.Engine.BasePage.ReadPage[T](PageBuffer buffer) at LiteDB.Engine.FileReaderV8.ReadPage[T](UInt32 pageID) at LiteDB.Engine.FileReaderV8.GetDocuments(String collection)+MoveNext() at LiteDB.Engine.LiteEngine.RebuildContent(IFileReader reader) at LiteDB.Engine.LiteEngine.Rebuild(RebuildOptions options) at LiteDB.LiteDatabase.Rebuild(RebuildOptions options)

When upgraded to LiteDB 5.0.11 I got this one when rebuilding System.Exception: LiteDB ENSURE: page type must be data page at LiteDB.Constants.ENSURE(Boolean conditional, String message) at LiteDB.Engine.DataPage..ctor(PageBuffer buffer) at LiteDB.Engine.BasePage.ReadPage[T](PageBuffer buffer) at LiteDB.Engine.FileReaderV8.ReadPage[T](UInt32 pageID) at LiteDB.Engine.FileReaderV8.GetDocuments(String collection)+MoveNext() at LiteDB.Engine.LiteEngine.RebuildContent(IFileReader reader) at LiteDB.Engine.LiteEngine.Rebuild(RebuildOptions options) at LiteDB.LiteDatabase.Rebuild(RebuildOptions options)

Some basic info:

database file size ~800MB
database log file ~78MB
most probably there was similar scenario, reading huge data & writing some data at the same time, maybe the disk was overloaded a while

Unfortunately I cannot share the database since it is customer one & password protected. Thank you for any update. Regards, Vlastimil

Jul 26 '21 09:07 meeety

Hi, I am getting this error in production too... It starts with the following exception:

System.Exception: LiteDB ENSURE: must have no pages in used when call Clear() cache
   at LiteDB.Constants.ENSURE(Boolean conditional, String message)
   at LiteDB.Engine.MemoryCache.Clear()
   at LiteDB.Engine.WalIndexService.Clear()
   at LiteDB.Engine.WalIndexService.CheckpointInternal()
   at LiteDB.Engine.WalIndexService.TryCheckpoint()
   at LiteDB.Engine.LiteEngine.CommitAndReleaseTransaction(TransactionService transaction)
   at LiteDB.Engine.LiteEngine.Commit()
   at LiteDB.LiteDatabase.Commit()

After that, the empty page exception occurs every time I try to insert more data:

System.Exception: LiteDB ENSURE: empty page must be defined as empty type
   at LiteDB.Constants.ENSURE(Boolean conditional, String message)
   at LiteDB.Engine.Snapshot.NewPage[T]()
   at LiteDB.Engine.Snapshot.GetFreeDataPage(Int32 bytesLength)
   at LiteDB.Engine.DataService.<>c__DisplayClass3_0.<<Insert>g__source|0>d.MoveNext()
   at LiteDB.Engine.BufferWriter..ctor(IEnumerable`1 source)
   at LiteDB.Engine.DataService.Insert(BsonDocument doc)
   at LiteDB.Engine.LiteEngine.InsertDocument(Snapshot snapshot, BsonDocument doc, BsonAutoId autoId, IndexService indexer, DataService data)
   at LiteDB.Engine.LiteEngine.<>c__DisplayClass7_0.<Insert>b__0(TransactionService transaction)
   at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn)
   at LiteDB.LiteCollection`1.Insert(T entity)

In our case we have set a data file size limit and use the direct connection mode. When the limit is reached, we delete the oldest data (basically a ring buffer). One thread is writing data to the data file periodically, and another thread is reading from it.

The problem starts when we delete old data. In MemoryCache.Clear() the PagesInUse is not 0, which means the ENSURE should throw an exception. I was never able to reproduce the error, or at least I thought so. When taking a closer look into the code I noticed the following code:

    /// <summary>
    /// Ensure condition is true, otherwise throw exception (check contract)
    /// </summary>
    [DebuggerHidden]
    public static void ENSURE(bool conditional, string message = null)
    {
      if (!conditional && !Debugger.IsAttached)
        throw new Exception("LiteDB ENSURE: " + message);
    }

The ENSURE actually fails for me too, but since I have my debugger attached, the error is just swallowed (see !Debugger.IsAttached) and no exception is thrown! The program silently continues in corrupted state.

@mbdavid is there any reason, why corrupted states are ignored when a debugger is attached?

Nov 03 '21 10:11 AppyxDaniel

Update: The problem for me seems to have been, that I was reading the documents from the collection, while removing those documents item by item after they have been read. Like this:

_Database.BeginTrans();

var toDeleteDocuments = _MyCollection.Find(**my condition**);

foreach (var deleteDocument in toDeleteDocuments)
{
    // Do some stuff with the 'deleteDocument'

    _MyCollection.Delete(deleteDocument.Id);
}
     
_Database.Commit();

Not using a transaction and deleting all documents at once after I am done processing the affected documents seems to have solved this issue. Like this:

var toDeleteDocuments = _DbValuePages.Find(**my condition**);
var toDeleteDocumentIds = new HashSet<long>();

foreach (var deleteDocument in toDeleteDocuments)
{
    // Do some stuff with the 'deleteDocument'


    // Remember ID of the document to delete.
    toDeleteDocumentIds.Add(deleteDocument.Id);
}

var tmp = toDeleteDocumentIds.Select(id => new BsonValue(id)).ToArray();
_MyCollection.DeleteMany(Query.In("_id", tmp));

Further I created a pull request to throw ENSURE exceptions when a debugger is attached as well. I am sure I am not the first person which had difficulties to reproduce the issue, because they use a debugger (https://github.com/mbdavid/LiteDB/pull/2094).

Nov 05 '21 10:11 AppyxDaniel

After running a rebuild a single time, I can't run it again due to this error.

[Exception: LiteDB ENSURE: page type must be collection page]
   LiteDB.Constants.ENSURE(Boolean conditional, String message) +97
   LiteDB.Engine.CollectionPage..ctor(PageBuffer buffer) +139
   LiteDB.Engine.BasePage.ReadPage(PageBuffer buffer) +269
   LiteDB.Engine.FileReaderV8.ReadPage(UInt32 pageID) +247
   LiteDB.Engine.<GetIndexes>d__6.MoveNext() +107
   LiteDB.Engine.LiteEngine.RebuildContent(IFileReader reader) +1123
   LiteDB.Engine.LiteEngine.Rebuild(RebuildOptions options) +890
   LiteDB.SharedEngine.Rebuild(RebuildOptions options) +56

I'm on LiteDB Nuget package version v5.0.11

Dec 15 '21 10:12 strike2867

Any resolution? Quick fix? App is crashing

Dec 17 '21 10:12 pooranoayaw

Sqlite?

Jan 31 '22 19:01 jjxtra

LiteDB LiteDB copied to clipboard

[BUG] System.Exception: LiteDB ENSURE: empty page must be defined as empty type

LiteDB
LiteDB copied to clipboard