Lucene.Net.Index.CorruptIndexException: invalid deletion count: 2 vs docCount=1

[konius]

Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)

11.3.1

Bug summary

Examine index gets corrupt and can't view or manage the Examine dashboard and any content trying to read index for display purpose becomes empty.

Happens on version 11.3.2, but also on 13.1.1 with the only solution to complete delete Examine folder and restart the application.

Issue is already discussed on Our.

Lucene.Net.Index.CorruptIndexException: invalid deletion count: 2 vs docCount=1 (resource: BufferedChecksumIndexInput(SimpleFSIndexInput(path="C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\MembersIndex\segments_vd")))
   at Lucene.Net.Index.SegmentInfos.Read(Directory directory, String segmentFileName)
   at Lucene.Net.Index.IndexFileDeleter..ctor(Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, InfoStream infoStream, IndexWriter writer, Boolean initialIndexExists)
   at Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf)
   at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
   at Examine.Lucene.Directories.DirectoryFactoryBase.<>c__DisplayClass2_0.<Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory>b__0(String s)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Examine.Lucene.Directories.DirectoryFactoryBase.Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
   at Umbraco.Cms.Infrastructure.Examine.ConfigurationEnabledDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
   at Examine.Lucene.Directories.DirectoryFactoryBase.<>c__DisplayClass2_0.<Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory>b__0(String s)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Examine.Lucene.Directories.DirectoryFactoryBase.Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
   at Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass1_0.<.ctor>b__0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Examine.Lucene.Providers.LuceneIndex.PerformIndexItemsInternal(IEnumerable`1 valueSets, CancellationToken cancellationToken)
   at Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass49_0.<PerformIndexItems>b__0()
   at Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass73_0.<QueueTask>b__0(Task x)

Specifics

For an unknown reason the index gets corrupt and bricks the back office dashboard.

Application is hosted on Azure and config is applied as per this guide: https://docs.umbraco.com/umbraco-cms/v/10.latest-lts/fundamentals/setup/server-setup/azure-web-apps

Steps to reproduce

N/A

Expected result / actual result

Expected to be able to at least view the dashboard and rebuild indexes if they get corrupt.

[github-actions]

Hi there @konius!

Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better.

We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.

• We'll assess whether this issue relates to something that has already been fixed in a later version of the release that it has been raised for.
• If it's a bug, is it related to a release that we are actively supporting or is it related to a release that's in the end-of-life or security-only phase?
• We'll replicate the issue to ensure that the problem is as described.
• We'll decide whether the behavior is an issue or if the behavior is intended.

We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions.

Thanks, from your friendly Umbraco GitHub bot 🤖 🙂

[shallett-ghd]

We've been dealing this issue with Umbraco 10.6.1. It has been impossible to replicate in development environments but does present itself in large traffic client sites with a lot of content.

[Migaroez]

Might be related to
#15783

[TQ-Benji]

Had similar issue on Umbraco Cloud using 10.8.5. Work around solution was to delete indexes via kudu and restart site - not ideal

[sytexa-julia]

Just started seeing this after we rolled out our upgrade to 13.x. Strangely it's been working fine in App Service for over a year.

[marius-ruhrmann-syzygy]

Same problem here.

• Hosted on Azure in windows environment
• Member index is corrupted
• Quick fix from @TQ-Benji works, but it would be cool to have a fix very soon!

[kows]

still happens on 13.3.0, single instance app service.

[readingdancer]

We have the same issue on a client site running on 10.8.5

One thing I thing that would be a great improvement would be for the Umbraco back office to actually respond when an error like this occurs. I did look into this the last time it happened and either the API just doesn't reply, or it replies with an error. Either way, the UI just continues to look like it's waiting to load the page. It would be much better if this was handled and an appropriate error message was displayed, like:

It looks like your indexes are _____ed ( broken ) , the best thing to do is log into Kudu and delete them all and restart your server, or if you are not technical, contact your technical contact and ask them to do it ( again ) for you.

Or words to that effect ;-)

[sniffdk]

We are running into this problem as well on a 13.1.0 installation running on a single Azure App Service.
The error seems to start at random, a deletion of the examine folder and a site restart fixes it, but not for long.

We can't even use the api to rebuild the index, as that will throw the same "invalid deletion count" error.

This should definitely have a higher priority in getting fixed 🙏

[benbarnett02]

Just adding my 2 cents - Same issue continuing on 13.3.2.

Running on Azure App Service (free plan), Azure SQL, Azure storage account for media & imagesharp stuff.

[Marcin-Niznik]

same problem Umbraco v12.3.10 on Umbraco Cloud, ContentDeliveryAPI Index gets corrupted.

[AdamKronquistToxic]

We have a similar problem on Umbraco 8, 10 and 13.
The website's search function stops working due to a corrupt index.

When searching

Response

Trying enter Examine Management

Logs for Examine Management

The solution is to delete the TEMP-folder in Kudo, restart the project and rebuild InternalIndex.

It's a quickfix but is not a long-term solution. For some of our customers, the problem recurs at weekly intervals.
1. Warning.txt
2. Error.txt
3. Error.txt
Error in Log Viewer when searching.txt

[sputnik-liam]

Experiencing this constantly running Umbraco 13.3.2 deployed to Azure using recommended config.
Lucene.Net.Index.CorruptIndexException: invalid deletion count: 171 vs docCount=1

It's especially troublesome as we don't have direct access to every customers infrastructure.

[seanrockster]

We're having this issue as well, it seems to be random. Had it a few weeks ago however removing the index files and restarting the webapp seemed to resolve it. However we're having the same issue. We've release some features and updates (same umbraco version 12.3.7) and we're suffering again. Removing the index files and restarting has had no affect.

Unfortunately our client relies on the member search because we have a custom index and searcher with member properties they need to search on - the standard searcher doesn't search on custom properties btw.

We're seeing different errors.

1. we're seeing the issue mentioned in this ticket
2. we're seeing log errors where examine is actually looking for filenames that do not exist, e.g. its looking for _1k.si and the membersindex folder doesn't have that file name, its _1q.si

System.IO.FileNotFoundException: Could not find file 'C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\MembersIndex\_1k.si'. File name: 'C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\MembersIndex\_1k.si' at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options) at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable1 unixCreateMode)
at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable1 unixCreateMode) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share) at Lucene.Net.Store.MMapDirectory.OpenInput(String name, IOContext context) at Lucene.Net.Store.Directory.OpenChecksumInput(String name, IOContext context) at Lucene.Net.Codecs.Lucene46.Lucene46SegmentInfoReader.Read(Directory dir, String segment, IOContext context) at Lucene.Net.Index.SegmentInfos.Read(Directory directory, String segmentFileName) at Lucene.Net.Index.SegmentInfos.FindSegmentsFileAnonymousClass.DoBody(String segmentFileName) at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run(IndexCommit commit) --- End of stack trace from previous location --- at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run(IndexCommit commit) at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run() at Lucene.Net.Index.SegmentInfos.Read(Directory directory) at Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf) at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock) at Examine.Lucene.Directories.DirectoryFactoryBase.<>c__DisplayClass2_0.<Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory>b__0(String s) at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory) at Examine.Lucene.Directories.DirectoryFactoryBase.Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock) at Umbraco.Cms.Infrastructure.Examine.ConfigurationEnabledDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock) at Examine.Lucene.Directories.DirectoryFactoryBase.<>c__DisplayClass2_0.<Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory>b__0(String s) at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory) at Examine.Lucene.Directories.DirectoryFactoryBase.Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock) at Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass1_0.<.ctor>b__0() at System.Lazy1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
at System.Lazy1.CreateValue() at Examine.Lucene.Providers.LuceneIndex.GetLuceneDirectory() at Examine.Lucene.Providers.LuceneIndex.IndexReady() at Examine.Lucene.Providers.LuceneIndex.PerformIndexItemsInternal(IEnumerable1 valueSets, CancellationToken cancellationToken)
at Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass49_0.b__0()
at Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass73_0.b__0(Task x)`

[seanrockster]

Deleting all of the examine indexes and restarting the webapp restored the indexes correct.

[sniffdk]

@seanrockster good to hear

But this seem to be a very common problem, would love to get some kind of HQ feedback here 😅

[sputnik-liam]

@seanrockster this is a known quick fix, but doesn't solve the underlying problem that has been a documented issue for a long time and theres been little to no formal communication from Umbraco on this.

[sniffdk]

Tagging @nul800sebastiaan 🙈

[authoritymikael]

We got this on Umraco Cloud and Cms 9.5.4. Deleting the examine indexes did it. But waiting for a permanent fix.

[AdamKronquistToxic]

We have experienced this on 8.18.14, 10.8.6 and 13.3.2.

[kevinstampe]

Also happening on 13.3.1, hosted in azure app service with azure sql db

EDIT:
Trying out this
Shazwazza/Examine#382 (comment)

[Migaroez]

Hey y'all, we are aware that these issues exist, but we have strong suspicions these things are popping up due to misconfiguration of azure web apps/load balancing/slot swapping. To be able to determine whether these issues are related to misconfiguration/bugs in examine/bugs in Umbraco, we are trying to build a troubleshooting guide, this will take some time still. In the meantime I advice you to read up on the issue @kevinstampe linked over on the examine repo (Shazwazza/Examine#382 (comment))
and the docs concerning azure/examine
https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/load-balancing/azure-web-apps#lucene-examine-configuration
https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/azure-web-apps#what-are-azure-web-apps

You can check a few of these configurations with @warrenbuckley's RuntimeValidators
https://github.com/Gibe/Umbraco.Community.RuntimeValidators

Hope to get back to you soon.

[sniffdk]

@Migaroez As @paulsterling writes here: Shazwazza/Examine#382 (comment) this is also an issue on a default Umbraco Cloud v. 13.4.0 configuration.
From your statement, that would mean, that the build-in configuration in Umbraco Cloud is faulty as well?

[eqtr-ab]

+1 for another site hosted in Umbraco Cloud encountering this same problem, Umbraco v13.3.2.

[mbogunovic]

Okay everyone, as @Shazwazza mentioned on Shazwazza/Examine#382. This could be due to:

1. Examine config being set to default here https://docs.umbraco.com/umbraco-cms/reference/configuration/examinesettings and we are all trying to use multi-instance load-balancer setup
2. I will try to apply and if I never come back to this post, means it works... 👯 "Examine": {
   "LuceneDirectoryFactory": "TempFileSystemDirectoryFactory"
   }
3. IN THEORY: This should indicate the examine to to create different cache path taking into an account hosting environment name so indexing doesn't get mixed up between two environment causing errors and inconsistencies.

Cheers!

[seanrockster]

We've upgraded to 13.4 and we are still having this issue. This is impacting our business, not good. @UmbracoHQ

[seanrockster]

Its logging this error every second, and the members index refuses to rebuild.

System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\MembersIndex\segments_3'.
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable1 unixCreateMode) at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable1 unixCreateMode)
at Lucene.Net.Store.Directory.Copy(Directory to, String src, String dest, IOContext context)
--- End of stack trace from previous location ---
at Lucene.Net.Store.Directory.Copy(Directory to, String src, String dest, IOContext context)
at Lucene.Net.Replicator.IndexReplicationHandler.RevisionReady(String version, IDictionary2 revisionFiles, IDictionary2 copiedFiles, IDictionary`2 sourceDirectory)
at Lucene.Net.Replicator.ReplicationClient.DoUpdate()
at Lucene.Net.Replicator.ReplicationClient.ReplicationThread.Run()

[Shazwazza]

Sure, i think if folks can update to 3.4.0 and see how it goes, than report back here. With any luck maybe we can close this in November.

[tommilleruk]

Any update on this? We're experiencing the issue on Umbraco Cloud CMS Version 12.2.0

[Shazwazza]

@tommilleruk yes, see above. Upgrade to examine 3.4.0 and tell us how it goes.

[somoreingold]

Just upgraded today, I'll report back if i see issues. So far the only thing to note is making sure to change any code that may be affected by the described breaking change (default result total decreased to 100 max).

[binraider]

I have just pushed the 3.4.0 upgrade to production and it has not fixed the problem. I am thinking of changing the target azure webapps from windows to linux to see if that helps

[tommilleruk]

@Shazwazza It seems that upgrading Examine hasn't fixed this based on others comments. We are running our website using Umbraco Cloud so I don't think it's something we would have configured incorrectly?

[Shazwazza]

@tommilleruk are you saying this from experience or just from what someone else has posted? It definitely has fixed this, or derivations of this because it proves it in the tests that were created and written in Examine.

If you are actively seeing this issue with Examine 3.4.0, please post a stack trace of the issue and verify that you definitely have the 3.4.0 dll deployed.

[mbogunovic]

@Shazwazza this was solved from me even before the patches (I left a comment some time ago) and now with the Examine 3.5 as well (tried 3.4 as well).

So I would say that this thread is now overloaded with misconfigurations on specific projects since there are few things to configure in order to work properly...

Cheers!

[Shazwazza]

@mbogunovic thanks a bunch for the feedback. Maybe @bergmania, @nul800sebastiaan, or others may resolve this ticket? If there are existing/future issues, will be best they are opened on their own.

[Shazwazza]

@markadrake stack trace please

[markadrake]

@Shazwazza I can supply some logs from Umbraco. I'll go ahead and collect the distinct entries.

This appears to be an issue with the indexes corrupting or SQL timeouts during indexing, but whatever the problem, I mistakenly read the OP's original error.

Here are some relevant logs; I'll continue adding them as I find distinct entries.

[PeterAtValantic]

@Shazwazza
I had this error last day as well.
I am on Umbraco 13.5.3 and have updated all nuget packages and Examine package is on 3.6.0.
I am hosting on Azure as well.

This is how my appsettings.json looks like:

"Umbraco": { "CMS": { "RuntimeMinification": { "UseInMemoryCache": true, }, "Global": { "MainDomLock": "FileSystemMainDomLock" }, "Hosting": { "LocalTempStorageLocation": "EnvironmentTemp" }, "Examine": { "LuceneDirectoryFactory": "SyncedTempFileSystemDirectoryFactory" } }

This is the umbraco log file:
{
"@t": "2025-02-03T11:55:04.0921069Z",
"@mt": "{IndexName} index is corrupt, a new one will be created",
"@l": "Error",
"@x": "Lucene.Net.Index.CorruptIndexException: invalid deletion count: 2 vs docCount=1 (resource: BufferedChecksumIndexInput(SimpleFSIndexInput(path="C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\InternalIndex\segments_1g1")))\r\n at Lucene.Net.Index.SegmentInfos.Read(Directory directory, String segmentFileName)\r\n at Lucene.Net.Index.IndexFileDeleter..ctor(Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, InfoStream infoStream, IndexWriter writer, Boolean initialIndexExists)\r\n at Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf)\r\n at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.GetIndexWriter(Directory mainDir, OpenMode openMode)\r\n at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.TryGetIndexWriter(OpenMode openMode, Directory luceneDirectory, Boolean createNewIfCorrupt, String indexName, IndexWriter& indexWriter)",
"IndexName": "InternalIndex",
"SourceContext": "Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory",
"ProcessId": 2532,
"ProcessName": "w3wp",
"ThreadId": 12,
"Log4NetLevel": "ERROR"
}

Another stacktrace of today:

invalid deletion count: 41 vs docCount=29 (resource: BufferedChecksumIndexInput(SimpleFSIndexInput(path="C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\ExternalIndex\segments_d5"))), Message: Handled in Application_Error (undefined url), ExMessage invalid deletion count: 41 vs docCount=29 (resource: BufferedChecksumIndexInput(SimpleFSIndexInput(path="C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\ExternalIndex\segments_d5"))), Ex: at Lucene.Net.Index.SegmentInfos.Read(Directory directory, String segmentFileName)
at Lucene.Net.Index.IndexFileDeleter..ctor(Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, InfoStream infoStream, IndexWriter writer, Boolean initialIndexExists)
at Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf)
at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.GetIndexWriter(Directory mainDir, OpenMode openMode)
at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.TryGetIndexWriter(OpenMode openMode, Directory luceneDirectory, Boolean createNewIfCorrupt, String indexName, IndexWriter& indexWriter)
at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.TryCreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock, Directory& directory)
at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
at Examine.Lucene.Directories.DirectoryFactoryBase.<>c__DisplayClass2_0.<Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory>b__0(String s)
at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory)
at Examine.Lucene.Directories.DirectoryFactoryBase.Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
at Umbraco.Cms.Infrastructure.Examine.ConfigurationEnabledDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
at Examine.Lucene.Directories.DirectoryFactoryBase.<>c__DisplayClass2_0.<Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory>b__0(String s)
at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory)
at Examine.Lucene.Directories.DirectoryFactoryBase.Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
at Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass1_0.<.ctor>b__0()
at System.Lazy1.ViaFactory(LazyThreadSafetyMode mode) at System.Lazy1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
at System.Lazy1.CreateValue() at System.Lazy1.get_Value()
at Examine.Lucene.Providers.LuceneIndex.GetLuceneDirectory()
at Examine.Lucene.Providers.LuceneIndex.IndexExistsImpl()
at Examine.Lucene.Providers.LuceneIndex.IndexExists()
at Examine.Lucene.Providers.LuceneIndex.EnsureIndex(Boolean forceOverwrite)
at Examine.Lucene.Providers.LuceneIndex.get_IndexWriter()
at Examine.Lucene.Providers.LuceneIndex.CreateSearcher()
at System.Lazy1.ViaFactory(LazyThreadSafetyMode mode) --- End of stack trace from previous location --- at System.Lazy1.CreateValue()
at Examine.Lucene.Providers.LuceneIndex.get_Searcher()
at SuperBold.Framework.UmbracoCustom.Services.SiteService.GetInitialQuery(String indexName, Boolean allowLeadingWildcard, Boolean onlyVisible)
at AspNetCore.Views_Partials_BlockList_Components_itemMainNavigation.ExecuteAsync() in C:\home\site\wwwroot\Views\Partials\BlockList\Components\itemMainNavigation.cshtml:line 56
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
at Microsoft.AspNetCore.Mvc.ViewFeatures.HtmlHelper.RenderPartialCoreAsync(String partialViewName, Object model, ViewDataDictionary viewData, TextWriter writer)
at Microsoft.AspNetCore.Mvc.ViewFeatures.HtmlHelper.PartialAsync(String partialViewName, Object model, ViewDataDictionary viewData)
at AspNetCore.Views_Partials_Blocklist_custom.ExecuteAsync() in C:\home\site\wwwroot\Views\Partials\Blocklist\custom.cshtml:line 18
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
at Microsoft.AspNetCore.Mvc.ViewFeatures.HtmlHelper.RenderPartialCoreAsync(String partialViewName, Object model, ViewDataDictionary viewData, TextWriter writer)
at Microsoft.AspNetCore.Mvc.ViewFeatures.HtmlHelper.PartialAsync(String partialViewName, Object model, ViewDataDictionary viewData)
at AspNetCore.Views_Partials_Menu.ExecuteAsync() in C:\home\site\wwwroot\Views\Partials\Menu.cshtml:line 93
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
at Microsoft.AspNetCore.Mvc.ViewFeatures.HtmlHelper.RenderPartialCoreAsync(String partialViewName, Object model, ViewDataDictionary viewData, TextWriter writer)
at Microsoft.AspNetCore.Mvc.ViewFeatures.HtmlHelper.PartialAsync(String partialViewName, Object model, ViewDataDictionary viewData)
at AspNetCore.Views_Master.<>c__DisplayClass34_0.<b__1>d.MoveNext() in C:\home\site\wwwroot\Views\Master.cshtml:line 103
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Razor.Runtime.TagHelpers.TagHelperExecutionContext.SetOutputContentAsync()
at AspNetCore.Views_Master.ExecuteAsync() in C:\home\site\wwwroot\Views\Master.cshtml:line 99
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderLayoutAsync(ViewContext context, ViewBufferTextWriter bodyWriter)
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable1 statusCode) at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable1 statusCode)
at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ActionContext actionContext, IView view, ViewDataDictionary viewData, ITempDataDictionary tempData, String contentType, Nullable`1 statusCode)
at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewResultExecutor.ExecuteAsync(ActionContext context, ViewResult result)
at Microsoft.AspNetCore.Mvc.ViewResult.ExecuteResultAsync(ActionContext context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Umbraco.Cms.Web.Common.Middleware.BasicAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Umbraco.Cms.Web.BackOffice.Middleware.BackOfficeExternalLoginProviderErrorMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at StackExchange.Profiling.MiniProfilerMiddleware.Invoke(HttpContext context) in C:\projects\dotnet\src\MiniProfiler.AspNetCore\MiniProfilerMiddleware.cs:line 112
at Umbraco.Cms.Web.Common.Middleware.UmbracoRequestMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Umbraco.Cms.Web.Common.Middleware.UmbracoRequestMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Umbraco.Cms.Web.Common.Middleware.PreviewAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Umbraco.Cms.Web.Common.Middleware.UmbracoRequestLoggingMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at SeoToolkit.Umbraco.Redirects.Core.Middleware.RedirectMiddleware.Invoke(HttpContext context)
at SeoToolkit.Umbraco.RobotsTxt.Core.Middleware.RobotsTxtMiddleware.Invoke(HttpContext context)
at SeoToolkit.Umbraco.Sitemap.Core.Middleware.SitemapMiddleware.Invoke(HttpContext context, ISitemapGenerator sitemapGenerator, ISitemapIndexGenerator sitemapIndexGenerator)
at SixLabors.ImageSharp.Web.Middleware.ImageSharpMiddleware.Invoke(HttpContext httpContext, Boolean retry)
at Umbraco.Forms.Web.HttpModules.ProtectFormUploadRequestsMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task).

So, how to solve this?

[Shazwazza]

@PeterAtValantic

So, how to solve this?

• Ensure that the Examine assembly deployed is in fact the latest version
• Open a new issue on the Examine repo https://github.com/shazwazza/Examine/
• Provide as much information as possible and potentially steps to replicate
  • Stack traces
  • What indexes get corrupt, 1, 2, all, etc...
  • Any relevant config

Then... we need to figure out how to replicate this in an isolated test setup environment - just like I did for this exact issue already, but clearly there's some other strange way to continue to break the indexes.

See https://github.com/Shazwazza/Examine/blob/support/3.x/src/Examine.Test/Examine.Lucene/Directories/SyncedFileSystemDirectoryFactoryTests.cs

@markadrake all of your errors appear to be related to "Lucene.Net.Store.LockObtainFailedException" for the DeliveryApiContentIndex. This is a new one I haven't seen but considering the location reported:

lockFile 'C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\DeliveryApiContentIndex\write.lock' alredy exists.

That location indicates that this index is not running on the local temp/fast drive and instead operating on the shared network drive. Please review your settings for "LuceneDirectoryFactory" for Azure hosting:

• https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/azure-web-apps
• https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/load-balancing/azure-web-apps#lucene-examine-configuration

[PeterAtValantic]

For the one following this thread: we continue here Shazwazza/Examine#402

[markadrake]

That location indicates that this index is not running on the local temp/fast drive and instead operating on the shared network drive. Please review your settings for "LuceneDirectoryFactory" for Azure hosting:

• https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/azure-web-apps
• https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/load-balancing/azure-web-apps#lucene-examine-configuration

Thank you for the info @Shazwazza! I'll give this a read through and see if there are details I can find specifically related to Umbraco Cloud. We happen to be using them to host this particular site. Thanks again mate!

[Shazwazza]

@PeterAtValantic actually now that I look further into your stack trace, this looks like the same problem as @markadrake...

That location indicates that this index is not running on the local temp/fast drive and instead operating on the shared network drive. Please review your settings for "LuceneDirectoryFactory" for Azure hosting:

https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/azure-web-apps
https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/load-balancing/azure-web-apps#lucene-examine-configuration

This is the path of your index:
C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\InternalIndex\

Which is not consistent with where it would exist if it were using the %temp% environment variable location.

[PeterAtValantic]

@Shazwazza

So what setting should I use then?

https://docs.umbraco.com/umbraco-cms/reference/configuration/examinesettings
Default - The index will operate from the default location: umbraco/Data/TEMP/ExamineIndexes

SyncedTempFileSystemDirectoryFactory - The index will operate on a local index created in the processes %temp% location and will replicate back to main storage in umbraco/Data/TEMP/ExamineIndexes

TempFileSystemDirectoryFactory - The index will operate only in the processes %temp% directory location

The one I use now is the "SyncedTempFileSystemDirectoryFactory".

Note: I am not on a load balanced server

I have followed the first link and my appsettings.json looks like: (with these settings I have the index issues)

"Umbraco": {
"CMS": {
"RuntimeMinification": {
"UseInMemoryCache": true,
},
"Global": {
"MainDomLock": "FileSystemMainDomLock"
},
"Hosting": {
"LocalTempStorageLocation": "EnvironmentTemp"
},
"Examine": {
"LuceneDirectoryFactory": "SyncedTempFileSystemDirectoryFactory"
}
}

This is our first project on Azure, so any tips and tricks about any config would be appreciated :)

[Shazwazza]

Thanks for the info @PeterAtValantic,

Sounds like its correct, if not load balancing and on azure then SyncedTempFileSystemDirectoryFactory is right. I realized that even though your stack trace shows files in the network file system, it could be because the directory also needs to open those files to do its replication. Happy to continue the chat/investigation on the Examine ticket. Note, i'm away next week so if I don't respond before then, that is why.

[protherj]

The one I use now is the "SyncedTempFileSystemDirectoryFactory".

Note: I am not on a load balanced server

Do you happen to use Deployment slots or have any other instances pointing to your live DB in any way (failover or blue/green) setup? That could mean that you technically are in a load balanced setup unintentionally. Not sure if that would have an effect in this case however, but could result in locks I think.

[PeterAtValantic]

Do you happen to use Deployment slots or have any other instances pointing to your live DB in any way (failover or blue/green) setup? That could mean that you technically are in a load balanced setup unintentionally. Not sure if that would have an effect in this case however, but could result in locks I think.

No, I use Visual Studio 2022 Publish to Azure to deploy the code and don't use any Deployment slots and the app is the only application who points to the Database.

[AdTarling]

We also experience this issue in our load balanced environment (Umbraco 13 currently, but seen on 7,9,10,13), albeit infrequently.
As per the load balanced docs (please see below) we have the scheduling publisher box (single instance) set as 'SyncedTempFileSystemDirectoryFactory' and subscriber box (load balanced instance) set as 'TempFileSystemDirectoryFactory'

The issue seems to occur sometimes when our overnight data importers that consume external API data and then need to update Umbraco's content interact with one of the scheduled publisher indexes, which seems to have got itself into an unrecoverable state. When this occurs I have to manually delete the indexes from the temp folder on the server and restart. Example log:
{"@t":"2025-03-09T02:25:04.9882276Z","@mt":"{IndexName} index is corrupt, a new one will be created","@l":"Error","@x":"Lucene.Net.Index.CorruptIndexException: invalid deletion count: 2 vs docCount=1 (resource: BufferedChecksumIndexInput(MMapIndexInput(path=\"C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\MembersIndex\\segments_28c\")))\r\n at Lucene.Net.Index.SegmentInfos.Read(Directory directory, String segmentFileName)\r\n at Lucene.Net.Index.IndexFileDeleter..ctor(Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, InfoStream infoStream, IndexWriter writer, Boolean initialIndexExists)\r\n at Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf)\r\n at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.GetIndexWriter(Directory mainDir, OpenMode openMode)\r\n at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.TryGetIndexWriter(OpenMode openMode, Directory luceneDirectory, Boolean createNewIfCorrupt, String indexName, IndexWriter& indexWriter)","IndexName":"MembersIndex","SourceContext":"Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory","ProcessId":8276,"ProcessName":"w3wp","ThreadId":29,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1xsdwk0008SF","Log4NetLevel":"ERROR"}

I have seen this on multiple installs on multiple versions of Umbraco, so has it ever been considered that if this occurs then there should be final line of defence coding to physically delete the Examine cache from the file system and force it to rebuild? I know that there is code in place to try and create a new index, but when this issue occurs, the underlying cache seems to be beyond repair so requires deleting and rebuilding.

[somoreingold]

We also experience this issue in our load balanced environment (Umbraco 13 currently, but seen on 7,9,10,13), albeit infrequently. As per the load balanced docs (please see below) we have the scheduling publisher box (single instance) set as 'SyncedTempFileSystemDirectoryFactory' and subscriber box (load balanced instance) set as 'TempFileSystemDirectoryFactory'

The issue seems to occur sometimes when our overnight data importers that consume external API data and then need to update Umbraco's content interact with one of the scheduled publisher indexes, which seems to have got itself into an unrecoverable state. When this occurs I have to manually delete the indexes from the temp folder on the server and restart. Example log: `{"@t":"2025-03-09T02:25:04.9882276Z","@mt":"{IndexName} index is corrupt, a new one will be created","@l":"Error","@x":"Lucene.Net.Index.CorruptIndexException: invalid deletion count: 2 vs docCount=1 (resource: BufferedChecksumIndexInput(MMapIndexInput(path=\"C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\MembersIndex\\segments_28c\")))\r\n at Lucene.Net.Index.SegmentInfos.Read(Directory directory, String segmentFileName)\r\n at Lucene.Net.Index.IndexFileDeleter..ctor(Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, InfoStream infoStream, IndexWriter writer, Boolean initialIndexExists)\r\n at Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf)\r\n at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.GetIndexWriter(Directory mainDir, OpenMode openMode)\r\n at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.TryGetIndexWriter(OpenMode openMode, Directory luceneDirectory, Boolean createNewIfCorrupt, String indexName, IndexWriter& indexWriter)","IndexName":"MembersIndex","SourceContext":"Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory","ProcessId":8276,"ProcessName":"w3wp","ThreadId":29,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1xsdwk0008SF","Log4NetLevel":"ERROR"}`

I have seen this on multiple installs on multiple versions of Umbraco, so has it ever been considered that if this occurs then there should be final line of defence coding to physically delete the Examine cache from the file system and force it to rebuild? I know that there is code in place to try and create a new index, but when this issue occurs, the underlying cache seems to be beyond repair so requires deleting and rebuilding.

I personally like this idea in general; we've had to do something similar to "self-heal" programmatically in this manner, but it would be nice if this last line of defense was more integrated in the package itself--if it makes sense and doesn't have unforeseen ripple effects.

[Shazwazza]

Good news everyone, Examine 3.7.0 has been released which should resolve this (again) https://github.com/Shazwazza/Examine/releases/tag/v3.7.0

[kristofferwebmind]

We installed version 3.7.0 and have had it on dev environment since March 20. Deployed to live environment on April 11 at 7 am, the site seemed fine during the day but it went down after business hours and the log is now again full of

Lucene.Net.Index.CorruptIndexException: invalid deletion count: 96 vs docCount=1 (resource: BufferedChecksumIndexInput(MMapIndexInput(path="C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\ExternalIndex\segments_2q4")))

[Shazwazza]

@kristofferwebmind tracked here Shazwazza/Examine#411
See comment Shazwazza/Examine#411 (comment)

[dawoe]

@Shazwazza I have upgraded to Examine 3.7.1 and don't see this issue anymore.

I also created a PR to update Examine in Umbraco : #19186