NuCache Out of Range Exception

[wadrschall]

Which exact Umbraco version are you using? For example: 9.0.1 - don't just write v9

10.0.1

Bug summary

After restarting the app in azure, the site would no longer load (500 errors including the umbraco backoffice). Upon reviewing the log files, the first error that is coming up indicates:

System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'handle')

Specifics

After restarting the app in azure, the site would no longer load (500 errors including the umbraco backoffice). Upon reviewing the log files, the first error that is coming up indicates:

System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'handle')

The code referenced in the error is this line that is called during our startup:

The full error from the log is:

System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'handle')
   at CSharpTest.Net.IO.TransactedCompoundFile.Read(UInt32 handle)
   at CSharpTest.Net.Storage.BTreeFileStoreV2.OpenRoot(Boolean& isNew)
   at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheNone.LoadStorage()
   at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheBase.Load()
   at CSharpTest.Net.Collections.BPlusTree`2..ctor(BPlusTreeOptions`2 ioptions)
   at CSharpTest.Net.Collections.BPlusTree`2..ctor(OptionsV2 optionsV2)
   at Umbraco.Cms.Infrastructure.PublishedCache.DataSource.BTree.GetTree(String filepath, Boolean exists, NuCacheSettings settings, ContentDataSerializer contentDataSerializer)
   at Umbraco.Cms.Infrastructure.PublishedCache.PublishedSnapshotService.MainDomRegister()
   at Umbraco.Cms.Core.Runtime.MainDom.Register(Action install, Action release, Int32 weight)
   at Umbraco.Cms.Infrastructure.PublishedCache.PublishedSnapshotService.<EnsureCaches>b__47_0()
   at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory)
   at System.Threading.LazyInitializer.EnsureInitialized[T](T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory)
   at Umbraco.Cms.Infrastructure.PublishedCache.PublishedSnapshotService.EnsureCaches()
   at Umbraco.Cms.Infrastructure.PublishedCache.PublishedSnapshotService.CreatePublishedSnapshot(String previewToken)
   at Umbraco.Cms.Web.Common.UmbracoContext.UmbracoContext.<>c__DisplayClass11_0.<.ctor>b__0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.get_Value()
   at Umbraco.Cms.Web.Common.UmbracoContext.UmbracoContext.get_PublishedSnapshot()
   at Umbraco.Cms.Web.Common.UmbracoContext.UmbracoContext.get_Content()
   at Capital.Web.Business.Routing.DirectoryListingMappings.MapDirectoryListingEndpoints(IEndpointRouteBuilder endpoints, ISiteSettingsService siteSettingsService, IUmbracoContextFactory umbracoContextFactory) in C:\\AzureDevOps\\agents\\Self Host\\BS-AGT-2\\_work\\1602\\s\\Capital.Web\\Business\\Routing\\DirectoryListingMappings.cs:line 22
   at Capital.Web.Startup.<>c__DisplayClass9_0.<Configure>b__3(IUmbracoEndpointBuilderContext u) in C:\\AzureDevOps\\agents\\Self Host\\BS-AGT-2\\_work\\1602\\s\\Capital.Web\\Startup.cs:line 303
   at Umbraco.Cms.Web.Common.ApplicationBuilder.UmbracoApplicationBuilder.<>c__DisplayClass17_0.<WithEndpoints>b__0(IEndpointRouteBuilder endpoints)
   at Microsoft.AspNetCore.Builder.EndpointRoutingApplicationBuilderExtensions.UseEndpoints(IApplicationBuilder builder, Action`1 configure)
   at Umbraco.Cms.Web.Common.ApplicationBuilder.UmbracoApplicationBuilder.WithEndpoints(Action`1 configureUmbraco)
   at Capital.Web.Startup.Configure(IApplicationBuilder app, IWebHostEnvironment env, ILoggerFactory loggerFactory, IDirectoryService directoryService, ICalendarService calendarService, ISiteSettingsService siteSettingsService, IConfiguration configuration, IUmbracoContextFactory umbracoContextFactory, ILogger`1 logger) in C:\\AzureDevOps\\agents\\Self Host\\BS-AGT-2\\_work\\1602\\s\\Capital.Web\\Startup.cs:line 290
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.ConfigureBuilder.Invoke(Object instance, IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.ConfigureBuilder.<>c__DisplayClass4_0.<Build>b__0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass15_0.<UseStartup>b__1(IApplicationBuilder app)
   at Microsoft.AspNetCore.Mvc.Filters.MiddlewareFilterBuilderStartupFilter.<>c__DisplayClass0_0.<Configure>g__MiddlewareFilterBuilder|0(IApplicationBuilder builder)
   at Microsoft.ApplicationInsights.AspNetCore.ApplicationInsightsStartupFilter.<>c__DisplayClass2_0.<Configure>b__0(IApplicationBuilder app)
   at Microsoft.AspNetCore.Server.IIS.Core.IISServerSetupFilter.<>c__DisplayClass2_0.<Configure>b__0(IApplicationBuilder app)
   at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)
   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Capital.Web.Program.Main(String[] args) in C:\\AzureDevOps\\agents\\Self Host\\BS-AGT-2\\_work\\1602\\s\\Capital.Web\\Program.cs:line 13

Steps to reproduce

Unknown how to replicate. Redeploying the code base (without any real functional change) and restarting the site made the error disappear. But this is concerning because this appears to be an error coming from an internal umbraco function that we don't have the ability to control and this is a site going live in a few weeks. We need to know if this is a stop/pause on deployment.

Expected result / actual result

Whatever this issue is (assuming its cache related), it shouldn't result in a full site outage and should be able to clear/reset any cache if necessary.

[github-actions[bot]]

Hi there @wadrschall!

Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better.

We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.

• We'll assess whether this issue relates to something that has already been fixed in a later version of the release that it has been raised for.
• If it's a bug, is it related to a release that we are actively supporting or is it related to a release that's in the end-of-life or security-only phase?
• We'll replicate the issue to ensure that the problem is as described.
• We'll decide whether the behavior is an issue or if the behavior is intended.

We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions.

Thanks, from your friendly Umbraco GitHub bot :robot: :slightly_smiling_face:

[nul800sebastiaan]

@wadrschall The last piece of code in the stack trace is:

at Capital.Web.Business.Routing.DirectoryListingMappings.MapDirectoryListingEndpoints(IEndpointRouteBuilder endpoints, ISiteSettingsService siteSettingsService, IUmbracoContextFactory umbracoContextFactory) in C:\\AzureDevOps\\agents\\Self Host\\BS-AGT-2\\_work\\1602\\s\\Capital.Web\\Business\\Routing\\DirectoryListingMappings.cs:line 22

Is that the code from your screenshot? Can you post the full code for DirectoryListingMappings, that might make it a lot clearer to see what you're doing there.

[nzdev]

Content likely exceeded block size, causing corruption.

[wadrschall]

@wadrschall The last piece of code in the stack trace is:

at Capital.Web.Business.Routing.DirectoryListingMappings.MapDirectoryListingEndpoints(IEndpointRouteBuilder endpoints, ISiteSettingsService siteSettingsService, IUmbracoContextFactory umbracoContextFactory) in C:\\AzureDevOps\\agents\\Self Host\\BS-AGT-2\\_work\\1602\\s\\Capital.Web\\Business\\Routing\\DirectoryListingMappings.cs:line 22

Is that the code from your screenshot? Can you post the full code for DirectoryListingMappings, that might make it a lot clearer to see what you're doing there.

Here's the file requested. It really doesn't do much except add some custom pattern matching routes. It looks up the URL to the Directory page so I have a page to start with for the detail pages, but that's the only call out to a DB. Long story short, the error is indicating a Nucache error, but there is nothing I'm doing here that should be custom or nucache related at this point.

DirectoryListingMappings.zip

[wadrschall]

Content likely exceeded block size, causing corruption.

Can you provide more detail about what you mean here? Where it says its failing is where I'm simply getting a root node. Why would that result in Nucache exceeding a block size? The root node itself has a handful of properties on it, but its not very large at all in the database. Is it possible that nucache is loosing track of something after a restart and the size it thinks it is isn't matching the actual size (or it has a size stored, but the value is no longer there)?

You also mention corruption as the issue here. If that was the case, there's no way this site comes back online. Restarting the site in Azure did not bring it back. Only by adding a space to one of the code files and deploying the site did it trigger this error to go away, which tells me its likely not a content issue, but rather some type of code issue that hit some case it couldn't account for and went into a failed state it couldn't come out of (even with a restart of the app, which also means it probably is something that's written to a file, etc as an app being stopped and restarted would have lost anything in memory, etc).

[nzdev]

Likely the redeploy worked because it deletes the nucache.content.db file from disk. When there is no file on disk, the sqldb is read into memory and stored on disk at startup. From there, only writes to the .db file occur. Next boot the files are there, but corrupted by the data exceeding the block size for one or more nodes. When the file is first accessed by using the published cache from code then that's when the out of range error occurs.

Options: Disable nucache disk cache. Increase the block size setting. Reduce the size of the content. More error handling so it's clear what this means. Rewrite nucache.

[kjac]

Hi @wadrschall --- just checking up here, are you still experiencing these issues on deploy?

[nikolajlauridsen]

Hey all, I'll have to go ahead and close down this issue, since we can't reproduce it and haven't heard back in 14 days, please feel free to let us know if you have additional information and then we can open up the issue again.