[BUG] The configured user limit (1024) on the number of inotify instances has been reached

[sdwerwed]

Describe the bug Once we deploy more than 3 .NET pods in AKS we get the following error

  Unhandled exception. System.IO.IOException: The configured user limit (1024) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
  at System.IO.FileSystemWatcher.StartRaisingEvents()
  at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
  at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
  at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
  at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
  at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
  at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
  at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
  at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
  at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
  at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
  at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()
  at Microsoft.Extensions.Hosting.HostBuilder.Build()
  at xxxxxxxxxxxxxxx.Platform.AuthServer.Program.Main(String[] args) in /src/xxxxxxxxxxxxxxx.Platform.AuthServer/Program.cs:line 14

Each pod consumes about 550 inotify instances.

Expected behavior I would expect to be able to create multiple .NET pods in the AKS or I would expect to be able to configure sysctl -w fs.inotify.max_user_instances=1048576 so I can create more than 3 pods per node.
I see in the official documentation it is supported fs.inotify.max_user_watches but not fs.inotify.max_user_instances.

Possible Solution Add the fs.inotify.max_user_instances in the Linux custom OS configuration settings.

Environment (please complete the following information):

• AKS 1.25.6

Workaround As a workaround, we have set a daemonset with root access to perform sysctl -w fs.inotify.max_user_instances=1048576. However, this solution has some implications, for example, pods will not start if daemonset will not be scheduled first, or it adds some vulnerabilities to the AKS as we do not want to run any pod with root access and is increasing the operational costs.

[dinfdsooff]

This is a blocker for us as well. fs.inotify.max_user_watches is supported but not fs.inotify.max_user_instances not. We would like this as soon as possible. related to this https://github.com/Azure/AKS/issues/772

[ghost]

Action required from @Azure/aks-pm

[thiDucTran]

we are also facing this issue

[ghost]

Issue needing attention of @Azure/aks-leads

[wangyira]

@justindavies could you help take a look

[Kenneth-Abrams]

We’re running into this exact problem as well. Can we get some attention to this issue and determine if it’s an issue with dotnet vs aks

[haitch]

@juan-lee

[allyford]

Updating this as a feature request for "Add the fs.inotify.max_user_instances in the Linux custom OS configuration settings".

[microsoft-github-policy-service[bot]]

This issue has been automatically marked as stale because it has not had any activity for 30 days. It will be closed if no further activity occurs within 7 days of this comment. @allyford

[microsoft-github-policy-service[bot]]

This issue will now be closed because it hasn't had any activity for 7 days after stale. sdwerwed feel free to comment again on the next 7 days to reopen or open a new issue after that time if you still have a question/issue or suggestion.