[BUG] The configured user limit (1024) on the number of inotify instances has been reached

[sdwerwed]

Describe the bug Once we deploy more than 3 .NET pods in AKS we get the following error

  Unhandled exception. System.IO.IOException: The configured user limit (1024) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
  at System.IO.FileSystemWatcher.StartRaisingEvents()
  at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
  at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
  at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
  at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
  at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
  at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
  at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
  at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
  at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
  at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()
  at Microsoft.Extensions.Hosting.HostBuilder.Build()
  at xxxxxxxxxxxxxxx.Platform.AuthServer.Program.Main(String[] args) in /src/xxxxxxxxxxxxxxx.Platform.AuthServer/Program.cs:line 14

Each pod consumes about 550 inotify instances.

Expected behavior I would expect to be able to create multiple .NET pods in the AKS or I would expect to be able to configure sysctl -w fs.inotify.max_user_instances=1048576 so I can create more than 3 pods per node.
I see in the official documentation it is supported fs.inotify.max_user_watches but not fs.inotify.max_user_instances.

Possible Solution Add the fs.inotify.max_user_instances in the Linux custom OS configuration settings.

Environment (please complete the following information):

• AKS 1.25.6

Workaround As a workaround, we have set a daemonset with root access to perform sysctl -w fs.inotify.max_user_instances=1048576. However, this solution has some implications, for example, pods will not start if daemonset will not be scheduled first, or it adds some vulnerabilities to the AKS as we do not want to run any pod with root access and is increasing the operational costs.