High CPU usage inside Docker Container

[georgedbarr]

Describe the bug

Hi all,

Noticed a potential bug in production where high CPU usage is causing the event loop to lock up. Chokidar is being run in a Docker Container inside a Kubernetes Cluster. This has been replicated in a local dev environment when running Chokidar with Node in a Docker Container.

It seems to be related to the number of files in the directory that is being watched, i.e. 10k files causes high CPU. Files size seems to not have an effect.

We are using polling as it is a network drive being watched.

Versions (please complete the following information):

• Chokidar version 3.4.3
• Node version: 12.15.0-alpine
• OS version: Alpine Linux

To Reproduce:

Config values:

usePolling: true,
awaitWriteFinish: {
  stabilityThreshold: 5000,
  pollInterval: 250,
}

Watcher:

this.watcher = chokidar
  .watch(this.config.foldersPaths, this.watcherConfig)
  .on('add', () => {})
  .on('change', () => {})
  .on('unlink', () => {})
  .on('ready', () => {})
  .on('error', error => {});

Start the docker container with no files in the directory. CPU usage is normal. Run the above command to generate files and CPU spikes and stays high. Changing the files size seems to have no effect, nor does pollInterval.

Restarting the service with the files already in the directory also causes the CPU usage to stay high.

Removing the files leads to the CPU decreasing back down to normal levels.

Expected behavior CPU usage to stay constant at low value.

Any help/ideas would be greatly appreciated!

[akmolina28]

Can confirm I am seeing this as well, in Docker for Windows as well as Debian.

[ElliotG]

Can confirm as well. It gets really bad at >500k files. Pretty much need a 3 cores dedicated to this problem. Would absolutely love an investment here.

[paulmillr]

@ElliotG if you'd "love an investment here", write some code for it. Or, "invest" into someone who writes code.

[ingles98]

Well I'm getting that with over 100k files on both linux docker container (on windows host) and directly on the host as well. Performance gets so bad with or without polling. With polling It goes a lot faster for the initial scan but uses a lot of resources and ends up hanging the app later on execution. Without polling it takes over 5 minutes for the initial scan and while it uses less resources it still ends up hanging the execution at some point.

[francescochiapello]

@georgedbarr have you found a solution ? I've the same issue watching about 25k nested files and folders. Thank you!

[ingles98]

@georgedbarr have you found a solution ? I've the same issue watching about 25k nested files and folders. Thank you!

I ended up reworking my FSW on top of NSFW (Node Sentinel Filesystem Watcher) and it works wonders.

Runs natively on Windows, Linux and MacOS. On windows it wraps around ReadDirectoryChangesW.

Currently only got one hurdle which actually seems to have been fixed 20~ days ago, though not yet updated on NPM.

It tends to throw CHANGED events upon LastAccessTime changes, and attributes as well. This can be worked around by keeping a Map of the filesystem and doing the stuff you want on CHANGED if the file actually changed based on reasonable factors (such as size and modifiedDate).