Race condition when watching dirs leads to missed files

[jimparis]

Describe the bug

There is a basic race condition when setting up a watch on a directory, in nodefs-handler.js: First the directory contents are scanned, and then a fs.watch watcher is set up. Any file created between those two operations won't be seen, until there's another operation that causes the fs.watch to trigger and the directory is re-scanned.

Versions (please complete the following information):

• 3.5.1
• 10.21.0
• Linux

To Reproduce:

First add this delay, which doesn't cause the race, just makes it a lot easier to trigger.

--- a/lib/nodefs-handler.js
+++ b/lib/nodefs-handler.js
@@ -556,6 +556,9 @@ async _handleDir(dir, stats, initialAdd, depth, target, wh, realpath) {
       if (this.fsw.closed) return;
     }
 
+    console.error("race!");
+    await new Promise(res => setTimeout(res, 5000));
+
     closer = this._watchWithNodeFs(dir, (dirPath, stats) => {
       // if current directory is removed, do nothing
       if (stats && stats.mtimeMs === 0) return;

Then use something like this

const chokidar = require('chokidar');
const fs = require('fs').promises;

function sleep(n) {
    return new Promise(res => setTimeout(res, n * 1000));
}

(async () => {
    try {
        const dir = await fs.mkdtemp('/tmp/foo-');

        const w = chokidar.watch(dir);
        w.on('all', (event, path) => {
            console.log("-- EVENT:", event, path);
        });
        w.on('ready', () => {
            console.log("-- EVENT: ready");
        });

        await sleep(2);
        console.log(`creating ${dir}/1`);
        await fs.writeFile(`${dir}/1`, "hi");

        await sleep(2);
        console.log(`creating ${dir}/2`);
        await fs.writeFile(`${dir}/2`, "hi");

        await sleep(2);
        console.log("nothing?");

        await sleep(10);
        console.log(`creating ${dir}/3`);
        await fs.writeFile(`${dir}/3`, "hi");

        await sleep(2);
        process.exit(0);
    } catch (e) {
        console.error(e);
    }
})();

Actual output

Events for adding files 1 and 2 don't happen until file 3 is created much later:

-- EVENT: addDir /tmp/foo-A2kUIi
race!
creating /tmp/foo-A2kUIi/1
creating /tmp/foo-A2kUIi/2
-- EVENT: ready
nothing?
creating /tmp/foo-A2kUIi/3
-- EVENT: add /tmp/foo-A2kUIi/1
-- EVENT: add /tmp/foo-A2kUIi/2
-- EVENT: add /tmp/foo-A2kUIi/3

Expected output

Events for adding files 1 and 2 should happen right before or after ready:

-- EVENT: addDir /tmp/foo-A2kUIi
race!
creating /tmp/foo-A2kUIi/1
creating /tmp/foo-oh0HPl/2
-- EVENT: ready
-- EVENT: add /tmp/foo-A2kUIi/1
-- EVENT: add /tmp/foo-A2kUIi/2
nothing?
creating /tmp/foo-A2kUIi/3
-- EVENT: add /tmp/foo-A2kUIi/3

[dividedmind]

Huh, I just spent a couple of hours tracking down the same exact issue. Serves me right for not searching first :)

I think I have a fix, a test case is more difficult however due to this being a race. The only sure way to cause the problem is to add that timeout in chokidar code.