chokidar
chokidar copied to clipboard
paulmillr
Glob failing to filter out symlinks that do not match the pattern in 1.6.1
What am I doing wrong here. I keep getting a 'add' event for every single file in a path I'm doing a watch on with a glob pattern.
var glob = path.resolve(watchDirectory) + "/**/*.dot";
console.log("Glob pattern: \n"+glob);
chokidar.watch(glob, {ignoreInitial: false}).on('all', (event, path) => {
console.log(event, path);
});
This outputs: (Notice the files being with 'add' events that don't have .dot extensions as per the glob pattern.
Glob pattern:
/Users/francoislaberge/dev/diagrams/**/*.dot
add /Users/francoislaberge/dev/diagrams/simple.dot
add /Users/francoislaberge/dev/diagrams/node_modules/.bin/electron
add /Users/francoislaberge/dev/diagrams/node_modules/electron-prebuilt/node_modules/.bin/extract-zip
add /Users/francoislaberge/dev/diagrams/node_modules/electron-prebuilt/node_modules/.bin/electron-download
add /Users/francoislaberge/dev/diagrams/node_modules/fsevents/node_modules/.bin/rc
add /Users/francoislaberge/dev/diagrams/node_modules/fsevents/node_modules/.bin/rimraf
add /Users/francoislaberge/dev/diagrams/node_modules/fsevents/node_modules/.bin/strip-json-comments
add /Users/francoislaberge/dev/diagrams/node_modules/fsevents/node_modules/.bin/har-validator
add /Users/francoislaberge/dev/diagrams/node_modules/fsevents/node_modules/.bin/mkdirp
add /Users/francoislaberge/dev/diagrams/node_modules/fsevents/node_modules/.bin/node-pre-gyp
add /Users/francoislaberge/dev/diagrams/node_modules/fsevents/node_modules/.bin/nopt
add /Users/francoislaberge/dev/diagrams/node_modules/fsevents/node_modules/.bin/semver
add /Users/francoislaberge/dev/diagrams/node_modules/fsevents/node_modules/.bin/sshpk-sign
add /Users/francoislaberge/dev/diagrams/node_modules/fsevents/node_modules/.bin/sshpk-conv
add /Users/francoislaberge/dev/diagrams/node_modules/fsevents/node_modules/.bin/sshpk-verify
add /Users/francoislaberge/dev/diagrams/node_modules/fsevents/node_modules/.bin/uuid
...
I'm experiencing exactly the same issue - I have Node v7.1.0 and chokidar 1.6.1 on MacOS Sierra. It looks like Chokidar already has known issues with Node 7 (see #550).
I'm seeing this with Node v5.12.0 and chokidar 1.6.1 on OS X El Capitan.
@francoislaberge it doesn't look like it's matching everything -- just files that actually match your glob pattern and files marked executable. You should change the title to "chokidar adds executable files that don't match glob"
I'm not sure if that's the case or if it was just picking up my binaries first before I cut off my copy/paste of the print out.
I no longer have this code setup anymore as I moved onto using another approach to work around it, otherwise I would verify if your theory is true. It certainly looks true from my report.
Should be mentioned that the executables are actually symlinks, and are handled by wh.filterDir, not wh.filterPath. They pass this, when they shouldn't. Unfortunately I have run out of time for debugging right now, but hopefully someone else can debug much quicker now that it's isolated.
[Edit] Issue is caused by using wh.filterDir for files. Since wh.filterDir only compares the directory parts of paths (and does so based on their index?), it ends up comparing "**" and "node_modules" - which passes. It completely ignores the last part of the pattern ("*.dot" in this case)
wh.filterDir itself is also broken when it comes to comparing directories. When the pattern is something like ./**/test/*.js, it'll fail to correctly identify ./foo/bar/test/baz.js as a match, because it compares by index. Is there any reason you don't simply strip the last part of the pattern, and compare the rest of the pattern to the dir path?
So was this issue introduced in chokidar 1.6.1? Can you check whether reverting down to 1.6.0 resolves it?
The symlinks going to filterDir instead of filterPath came from https://github.com/paulmillr/chokidar/issues/530, but that solution may not have been vetted enough before going out.
Your last point about there being better ways to handle filterDir - sure, probably. The intention of that function is to tell readdirp whether to recurse into a directory or not, so it's intended to only rule things out very conservatively because the worst case is drilling down into dirs where none of the entries end up matching. You can see there's a globstar (**) short-circuit in there for situations like the one you described. But if you identify a better way, PR welcome.
Yes, reverting to 1.6.0 does resolve the issue.
I believe the proper way to fix this would be to check if the realpath that the symlink points to is a directory/file, and then use the appropriate filtering method rather than simply defaulting to dir filtering when a symlink is encountered.
Given that the filtering functions are used by readdirp, I don't think async is even supported. It'd have to use the sync variants either way, which greatly simplifies the implementation.
Avoiding any sync fs operations has been a cardinal rule here, but if there's no choice, then I guess that'll just be the penalty that has to be paid for symlinks.
But perhaps these should just not be filtered at the readdirp level and get handled elsewhere.
@birjolaxew wait, so if there's a directory symlink e.g. builds/latest -> builds/0.9.20, would files inside it match builds/latest/*, builds/0.9.20/*, or both? I would think all paths scanned should be tested against the glob(s) as-is, without resolving symlinks.
@jedwards1211 A symlink is essentially the same thing as having a directory/file in two places. If you have a directory latest which is a symlink to 0.9.20, then any files inside 0.9.20 also exist inside latest. If you watch both folders, you'll get two "changed" events every time a file inside 0.9.20 changes - because it's essentially two files being changed. If you only watch one, you'll only get the "changed" event related to that folder.
The files inside latest won't match the pattern 0.9.20/*, but they'll be updated at the same time as those in 0.9.20 (because that's what symlinks do).
The issue here is that symlinks can also be files, while chokidar assumes that all symlinks are folders. This means file symlinks (like those in node_modules/.bin/) are filtered based on directory rules, not file rules.
@birjolaxew yes I understand how symlinks work. I just wanted to make sure that it wasn't filtering on paths with all symlinks resolved out, because that would be a major problem.
Ideally chokidar should handle symlinks in exactly the same way as multiple hardlinks to the same file/directory, right?
@jedwards1211 Yes. Chokidar will handle symlink as if though they were folders/files - except for the issue described here.
I just ran into this as well. I suggest 1.6.1 be reverted for now to fix the regression. https://github.com/paulmillr/chokidar/commit/9eeecf605ba24798d90e93206d6867d8469320f7
I just encountered this problem in 3.0.2. My system environment is macOS 10.14.5, because I use gulp in the project, gulp depends on chokidar, ~~I can only ignore all symlink files by followSymlinks : false now~~(don't work). I can't directly lower the version of chokidar.
If someone can make a reproducible repository with the issue, thay would help.
