Lucene.Net.Index.CorruptIndexException: invalid deletion count: 2 vs docCount=1

[konius]

Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)

11.3.1

Bug summary

Examine index gets corrupt and can't view or manage the Examine dashboard and any content trying to read index for display purpose becomes empty.

Happens on version 11.3.2, but also on 13.1.1 with the only solution to complete delete Examine folder and restart the application.

Issue is already discussed on Our.

Lucene.Net.Index.CorruptIndexException: invalid deletion count: 2 vs docCount=1 (resource: BufferedChecksumIndexInput(SimpleFSIndexInput(path="C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\MembersIndex\segments_vd")))
   at Lucene.Net.Index.SegmentInfos.Read(Directory directory, String segmentFileName)
   at Lucene.Net.Index.IndexFileDeleter..ctor(Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, InfoStream infoStream, IndexWriter writer, Boolean initialIndexExists)
   at Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf)
   at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
   at Examine.Lucene.Directories.DirectoryFactoryBase.<>c__DisplayClass2_0.<Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory>b__0(String s)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Examine.Lucene.Directories.DirectoryFactoryBase.Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
   at Umbraco.Cms.Infrastructure.Examine.ConfigurationEnabledDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
   at Examine.Lucene.Directories.DirectoryFactoryBase.<>c__DisplayClass2_0.<Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory>b__0(String s)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Examine.Lucene.Directories.DirectoryFactoryBase.Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
   at Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass1_0.<.ctor>b__0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Examine.Lucene.Providers.LuceneIndex.PerformIndexItemsInternal(IEnumerable`1 valueSets, CancellationToken cancellationToken)
   at Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass49_0.<PerformIndexItems>b__0()
   at Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass73_0.<QueueTask>b__0(Task x)

Specifics

For an unknown reason the index gets corrupt and bricks the back office dashboard.

Application is hosted on Azure and config is applied as per this guide: https://docs.umbraco.com/umbraco-cms/v/10.latest-lts/fundamentals/setup/server-setup/azure-web-apps

Steps to reproduce

N/A

Expected result / actual result

Expected to be able to at least view the dashboard and rebuild indexes if they get corrupt.

[github-actions[bot]]

Hi there @konius!

Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better.

We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.

• We'll assess whether this issue relates to something that has already been fixed in a later version of the release that it has been raised for.
• If it's a bug, is it related to a release that we are actively supporting or is it related to a release that's in the end-of-life or security-only phase?
• We'll replicate the issue to ensure that the problem is as described.
• We'll decide whether the behavior is an issue or if the behavior is intended.

We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions.

Thanks, from your friendly Umbraco GitHub bot :robot: :slightly_smiling_face:

[shallett-ghd]

We've been dealing this issue with Umbraco 10.6.1. It has been impossible to replicate in development environments but does present itself in large traffic client sites with a lot of content.

[Migaroez]

Might be related to https://github.com/umbraco/Umbraco-CMS/issues/15783

[TQ-Benji]

Had similar issue on Umbraco Cloud using 10.8.5. Work around solution was to delete indexes via kudu and restart site - not ideal

[sytexa-julia]

Just started seeing this after we rolled out our upgrade to 13.x. Strangely it's been working fine in App Service for over a year.

[marius-ruhrmann-syzygy]

Same problem here.

• Hosted on Azure in windows environment
• Member index is corrupted
• Quick fix from @TQ-Benji works, but it would be cool to have a fix very soon!

[kows]

still happens on 13.3.0, single instance app service.

[readingdancer]

We have the same issue on a client site running on 10.8.5

One thing I thing that would be a great improvement would be for the Umbraco back office to actually respond when an error like this occurs. I did look into this the last time it happened and either the API just doesn't reply, or it replies with an error. Either way, the UI just continues to look like it's waiting to load the page. It would be much better if this was handled and an appropriate error message was displayed, like:

It looks like your indexes are _____ed ( broken ) , the best thing to do is log into Kudu and delete them all and restart your server, or if you are not technical, contact your technical contact and ask them to do it ( again ) for you.

Or words to that effect ;-)

[sniffdk]

We are running into this problem as well on a 13.1.0 installation running on a single Azure App Service. The error seems to start at random, a deletion of the examine folder and a site restart fixes it, but not for long.

We can't even use the api to rebuild the index, as that will throw the same "invalid deletion count" error.

This should definitely have a higher priority in getting fixed 🙏

[benbarnett02]

Just adding my 2 cents - Same issue continuing on 13.3.2.

Running on Azure App Service (free plan), Azure SQL, Azure storage account for media & imagesharp stuff.

[Marcin-Niznik]

same problem Umbraco v12.3.10 on Umbraco Cloud, ContentDeliveryAPI Index gets corrupted.

[AdamKronquistToxic]

We have a similar problem on Umbraco 8, 10 and 13. The website's search function stops working due to a corrupt index.

When searching

Response

Trying enter Examine Management

Logs for Examine Management

The solution is to delete the TEMP-folder in Kudo, restart the project and rebuild InternalIndex.

It's a quickfix but is not a long-term solution. For some of our customers, the problem recurs at weekly intervals. 1. Warning.txt 2. Error.txt 3. Error.txt Error in Log Viewer when searching.txt

[sputnik-liam]

Experiencing this constantly running Umbraco 13.3.2 deployed to Azure using recommended config. Lucene.Net.Index.CorruptIndexException: invalid deletion count: 171 vs docCount=1

It's especially troublesome as we don't have direct access to every customers infrastructure.

[seanrockster]

We're having this issue as well, it seems to be random. Had it a few weeks ago however removing the index files and restarting the webapp seemed to resolve it. However we're having the same issue. We've release some features and updates (same umbraco version 12.3.7) and we're suffering again. Removing the index files and restarting has had no affect.

Unfortunately our client relies on the member search because we have a custom index and searcher with member properties they need to search on - the standard searcher doesn't search on custom properties btw.

We're seeing different errors.

1. we're seeing the issue mentioned in this ticket
2. we're seeing log errors where examine is actually looking for filenames that do not exist, e.g. its looking for _1k.si and the membersindex folder doesn't have that file name, its _1q.si

System.IO.FileNotFoundException: Could not find file 'C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\MembersIndex\_1k.si'. File name: 'C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\MembersIndex\_1k.si' at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options) at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable1 unixCreateMode) at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable1 unixCreateMode) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share) at Lucene.Net.Store.MMapDirectory.OpenInput(String name, IOContext context) at Lucene.Net.Store.Directory.OpenChecksumInput(String name, IOContext context) at Lucene.Net.Codecs.Lucene46.Lucene46SegmentInfoReader.Read(Directory dir, String segment, IOContext context) at Lucene.Net.Index.SegmentInfos.Read(Directory directory, String segmentFileName) at Lucene.Net.Index.SegmentInfos.FindSegmentsFileAnonymousClass.DoBody(String segmentFileName) at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run(IndexCommit commit) --- End of stack trace from previous location --- at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run(IndexCommit commit) at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run() at Lucene.Net.Index.SegmentInfos.Read(Directory directory) at Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf) at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock) at Examine.Lucene.Directories.DirectoryFactoryBase.<>c__DisplayClass2_0.<Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory>b__0(String s) at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory) at Examine.Lucene.Directories.DirectoryFactoryBase.Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock) at Umbraco.Cms.Infrastructure.Examine.ConfigurationEnabledDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock) at Examine.Lucene.Directories.DirectoryFactoryBase.<>c__DisplayClass2_0.<Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory>b__0(String s) at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory) at Examine.Lucene.Directories.DirectoryFactoryBase.Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock) at Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass1_0.<.ctor>b__0() at System.Lazy1.ViaFactory(LazyThreadSafetyMode mode) --- End of stack trace from previous location --- at System.Lazy1.CreateValue() at Examine.Lucene.Providers.LuceneIndex.GetLuceneDirectory() at Examine.Lucene.Providers.LuceneIndex.IndexReady() at Examine.Lucene.Providers.LuceneIndex.PerformIndexItemsInternal(IEnumerable1 valueSets, CancellationToken cancellationToken) at Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass49_0.<PerformIndexItems>b__0() at Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass73_0.<QueueTask>b__0(Task x)`

[seanrockster]

Deleting all of the examine indexes and restarting the webapp restored the indexes correct.

[sniffdk]

@seanrockster good to hear

But this seem to be a very common problem, would love to get some kind of HQ feedback here 😅

[sputnik-liam]

@seanrockster this is a known quick fix, but doesn't solve the underlying problem that has been a documented issue for a long time and theres been little to no formal communication from Umbraco on this.

[sniffdk]

Tagging @nul800sebastiaan 🙈

[authoritymikael]

We got this on Umraco Cloud and Cms 9.5.4. Deleting the examine indexes did it. But waiting for a permanent fix.

[AdamKronquistToxic]

We have experienced this on 8.18.14, 10.8.6 and 13.3.2.

[kevinstampe]

Also happening on 13.3.1, hosted in azure app service with azure sql db

EDIT: Trying out this https://github.com/Shazwazza/Examine/issues/382#issuecomment-2158962439

[Migaroez]

Hey y'all, we are aware that these issues exist, but we have strong suspicions these things are popping up due to misconfiguration of azure web apps/load balancing/slot swapping. To be able to determine whether these issues are related to misconfiguration/bugs in examine/bugs in Umbraco, we are trying to build a troubleshooting guide, this will take some time still. In the meantime I advice you to read up on the issue @kevinstampe linked over on the examine repo (https://github.com/Shazwazza/Examine/issues/382#issuecomment-2158962439) and the docs concerning azure/examine https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/load-balancing/azure-web-apps#lucene-examine-configuration https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/azure-web-apps#what-are-azure-web-apps

You can check a few of these configurations with @warrenbuckley's RuntimeValidators https://github.com/Gibe/Umbraco.Community.RuntimeValidators

Hope to get back to you soon.

[sniffdk]

@Migaroez As @paulsterling writes here: https://github.com/Shazwazza/Examine/issues/382#issuecomment-2189532338 this is also an issue on a default Umbraco Cloud v. 13.4.0 configuration. From your statement, that would mean, that the build-in configuration in Umbraco Cloud is faulty as well?

[eqtr-ab]

+1 for another site hosted in Umbraco Cloud encountering this same problem, Umbraco v13.3.2.

[mbogunovic]

Okay everyone, as @Shazwazza mentioned on https://github.com/Shazwazza/Examine/issues/382. This could be due to:

1. Examine config being set to default here https://docs.umbraco.com/umbraco-cms/reference/configuration/examinesettings and we are all trying to use multi-instance load-balancer setup
2. I will try to apply and if I never come back to this post, means it works... 👯 "Examine": { "LuceneDirectoryFactory": "TempFileSystemDirectoryFactory" }
3. IN THEORY: This should indicate the examine to to create different cache path taking into an account hosting environment name so indexing doesn't get mixed up between two environment causing errors and inconsistencies.

Cheers!

[seanrockster]

We've upgraded to 13.4 and we are still having this issue. This is impacting our business, not good. @UmbracoHQ

[seanrockster]

Its logging this error every second, and the members index refuses to rebuild.

System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\MembersIndex\segments_3'. at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable1 unixCreateMode) at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable1 unixCreateMode) at Lucene.Net.Store.Directory.Copy(Directory to, String src, String dest, IOContext context) --- End of stack trace from previous location --- at Lucene.Net.Store.Directory.Copy(Directory to, String src, String dest, IOContext context) at Lucene.Net.Replicator.IndexReplicationHandler.RevisionReady(String version, IDictionary2 revisionFiles, IDictionary2 copiedFiles, IDictionary`2 sourceDirectory) at Lucene.Net.Replicator.ReplicationClient.DoUpdate() at Lucene.Net.Replicator.ReplicationClient.ReplicationThread.Run()

[Shazwazza]

@seanrockster please see this for now and do use the different factory as a work around for now https://github.com/Shazwazza/Examine/issues/382#issuecomment-2163520075

There's been some work on this recently to try to address all of this, but will require a beta release and some testing (next week). In the meantime, you should use UmbracoTempEnvFileSystemDirectoryFactory

[seanrockster]

@Shazwazza Ok tried that, causes the app to restart and now I get HTTP Error 500.30 - ASP.NET Core app failed to start. OMG this is a high volume ecommerce site, it cannot be donw.

[seanrockster]

@Shazwazza that didn't work and brought the site down with some error - i'll check the log for the exact error but had to revert to TempFileSystemDirectoryFactory