[BUG] Detected loop in FindAll({0})

[kamilwarowny]

Version .Net 8.0 LiteDB.dll 5.0.21.0

Describe the bug About 16000 records in file. File size 33MB

Code to Reproduce

using (var db = new LiteDatabase(dbFilePAth))
{                   
    var col = db.GetCollection<UserHash>(CollectionName);

    var userHashes = col.Query().ToList();

    db.Checkpoint();

    db.Rebuild();

    return userHashes;
}

line db.Rebuild(); throws error below:

at LiteDB.Engine.IndexService.FindAll(CollectionIndex index, Int32 order)+MoveNext()
at LiteDB.LinqExtensions.<>c__DisplayClass2_02.<<DistinctBy>g___|0>d.MoveNext() at LiteDB.Engine.LiteEngine.<>c__DisplayClass5_0.<EnsureIndex>b__0(TransactionService transaction) at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func2 fn)
at LiteDB.Engine.LiteEngine.EnsureIndex(String collection, String name, BsonExpression expression, Boolean unique)
at LiteDB.Engine.LiteEngine.RebuildContent(IFileReader reader)
at LiteDB.Engine.RebuildService.Rebuild(RebuildOptions options)
at LiteDB.Engine.LiteEngine.Rebuild(RebuildOptions options)
at LiteDB.LiteDatabase.Rebuild(RebuildOptions options)

[holychaos]

I had this problem too. I tried to move the 'db' file out of the original path, run the program once to generate a new 'db' file, and finally overwrite the moved 'db' file back, and everything worked.

[marss72]

Just BTW, not regarding your issue, but it might be possible to improve the provided code by maybe better performance and clarity.

If it's really necessary to use List<T>, ignore this one. But you can improve memory efficiency and lazily load all the entries. I am not an expert on LiteDB so this might cause issues since you are rebuilding it right after.

Also, you can modify col.Query().ToList(); to use FindAll() instead of the Query(). Try it yourself, you might get a small performance benefit or not, but I haven't measured it.

[k0ka]

I have the same problem. It happens if any table on rebuild has more than 2550 records. Here is why:

1. RedbuildService creates a new database https://github.com/mbdavid/LiteDB/blob/c4db4ab3b1e594439921d594f403775c82f9e8c7/LiteDB/Engine/Services/RebuildService.cs#L55
2. It starts rebuilding it using the main IFileReader https://github.com/mbdavid/LiteDB/blob/c4db4ab3b1e594439921d594f403775c82f9e8c7/LiteDB/Engine/Engine/Rebuild.cs#L52
3. When it creates IndexService it passes _disk.MAX_ITEM_COUNT from the newly created database. Which means it is always minimum value of 2550. https://github.com/mbdavid/LiteDB/blob/c4db4ab3b1e594439921d594f403775c82f9e8c7/LiteDB/Engine/Engine/Rebuild.cs#L63

I'm not sure how to properly fix it. The easiest way would be to just pass the correct MAX_ITEM_COUNT from the RebuildService where it still knows this value.

[zalza13]

This property was added in version 5.0.019 and I believe it is causing that calculation to be incorrect.

    /// <summary>
    /// This method calculates the maximum number of items (documents or IndexNodes) that this database can have.
    /// The result is used to prevent infinite loops in case of problems with pointers
    /// Each page support max of 255 items. Use 10 pages offset (avoid empty disk)
    /// </summary>
    public uint MAX_ITEMS_COUNT => (uint)(((_dataLength + _logLength) / PAGE_SIZE) + 10) * byte.MaxValue;

In my case I have a collection with 5000+ documents and my application is collapsing.

Possibles workaround:

1. Go back to the version 5.0.18
2. Comment the lines ENSURE(counter++ < _maxItemsCount, "Detected loop in GetNodeList({0})" (I guess no a good idea)
3. Wait for a fix

@mbdavid I know you are not involved in the project anymore and that's great, but could you at least give us a hint on how to solve it or how we can at least apply a temporary solution? --> https://github.com/litedb-org/LiteDB/commit/663f7490feb78ee28968abb27d72e9537ddcb4b4

Thank you so much!! :D

[patryk9200]

I've published a temporary nuget package variant with fix for this bug: https://www.nuget.org/packages/LiteDB.Fixed/5.0.22

The source code: https://github.com/patryk9200/LiteDB