node-sync-glob icon indicating copy to clipboard operation
node-sync-glob copied to clipboard
verified publisher icon AndyOGo

Subdirectories not deleted during `sync-glob --watch` on Windows 10 / Linux

Open KeithGillette opened this issue 8 years ago • 18 comments

When running:

sync-glob --watch 'source/*' destination or: sync-glob --watch 'source/**/*' destination

directories and files are copied and updated (recursively) from source to destination, but the watch does not delete subdirectories deleted in source during the watch.

With:

  • sync-glob 1.3.6
  • npm 3.10.10
  • Node 6.9.5
  • Windows_NT 10.0.14393

KeithGillette avatar Mar 30 '17 17:03 KeithGillette

Thanks for the fast patching, @AndyOGo! Your release 1.3.7 did fix issue #41, so files are now correctly deleted, but in my test it still does not detect (empty) subdirectory deletion on Windows 10.

KeithGillette avatar Mar 30 '17 22:03 KeithGillette

@KeithGillette Thanks for your quick feedback. I will investigate this issue. From your data it seems have a missing test case too.

AndyOGo avatar Mar 31 '17 11:03 AndyOGo

I just run the following test on my Mac Note: all sub directories here are not empty.

node bin/sync-glob.js --watch 'test/**/*' tmp/

Initial Copy

DELETE tmp COPY test/mock/bar to tmp/bar COPY test/mock/foo to tmp/foo COPY test/mock/@org to tmp/@org COPY test/mock/a.txt to tmp/a.txt COPY test/mock/b.txt to tmp/b.txt COPY test/mock/transform.js to tmp/transform.js COPY test/mock/bar/c.txt to tmp/bar/c.txt COPY test/mock/foo/b.txt to tmp/foo/b.txt COPY test/mock/foo/d.txt to tmp/foo/d.txt COPY test/mock/@org/d.txt to tmp/@org/d.txt COPY test/mock/@org/b.txt to tmp/@org/b.txt COPY test/mock/@org/a.txt to tmp/@org/a.txt WATCHING test/mock/**/*

Deleting test/mock/ sub directory

DELETE tmp/mock/@org/a.txt DELETE tmp/mock/@org/b.txt DELETE tmp/mock/@org/d.txt DELETE tmp/mock/@org DELETE tmp/mock/bar/c.txt DELETE tmp/mock/bar DELETE tmp/mock/foo/b.txt DELETE tmp/mock/foo/d.txt DELETE tmp/mock/foo DELETE tmp/mock/b.txt DELETE tmp/mock/a.txt DELETE tmp/mock/transform.js DELETE tmp/mock

Deleting test/mock/@org/ sub sub directory

DELETE tmp/mock/@org/b.txt DELETE tmp/mock/@org/a.txt DELETE tmp/mock/@org/d.txt DELETE tmp/mock/@org

AndyOGo avatar Mar 31 '17 20:03 AndyOGo

So double I just created an empty sub directory and run following test:

node bin/sync-glob.js --watch 'test/**/*' tmp/

DELETE tmp COPY test/lib to tmp/lib COPY test/mock to tmp/mock COPY test/mock/@org to tmp/mock/@org COPY test/mock/bar to tmp/mock/bar COPY test/mock/empty to tmp/mock/empty COPY test/mock/foo to tmp/mock/foo COPY test/copy.spec.js to tmp/copy.spec.js COPY test/helpers.js to tmp/helpers.js COPY test/lib/is-glob.spec.js to tmp/lib/is-glob.spec.js COPY test/lib/resolve-target.spec.js to tmp/lib/resolve-target.spec.js COPY test/lib/sources-bases.spec.js to tmp/lib/sources-bases.spec.js COPY test/lib/trim-quotes.spec.js to tmp/lib/trim-quotes.spec.js COPY test/mock/@org/a.txt to tmp/mock/@org/a.txt COPY test/mock/@org/b.txt to tmp/mock/@org/b.txt COPY test/mock/@org/d.txt to tmp/mock/@org/d.txt COPY test/mock/a.txt to tmp/mock/a.txt COPY test/mock/b.txt to tmp/mock/b.txt COPY test/mock/bar/c.txt to tmp/mock/bar/c.txt COPY test/mock/foo/b.txt to tmp/mock/foo/b.txt COPY test/mock/foo/d.txt to tmp/mock/foo/d.txt COPY test/mock/transform.js to tmp/mock/transform.js COPY test/sync.spec.js to tmp/sync.spec.js COPY test/transform.spec.js to tmp/transform.spec.js WATCHING test/**/*

Delete test/mock/empty by running rmdir test/mock/empty

DELETE tmp/mock/empty

@KeithGillette Seems all okay on OSX. May I asked you if you get any errors? Or could you post your logs please? Unfortunately I don't have a Windows 10 machine available 😞 And please use the --verbose option.

AndyOGo avatar Mar 31 '17 20:03 AndyOGo

@KeithGillette I quickly googled if chokidar has a deletion issue. And indeed it describes exactly your problem with following error:

Error: watch null EPERM

  • Still open: https://github.com/paulmillr/chokidar/issues/566
  • Already closed, but related: https://github.com/paulmillr/chokidar/issues/422 https://github.com/paulmillr/chokidar/issues/88

AndyOGo avatar Mar 31 '17 20:03 AndyOGo

@KeithGillette Can you confirm that you have an permission error?

AndyOGo avatar Mar 31 '17 20:03 AndyOGo

Hi @AndyOGo. I get absolutely no errors in the console even with the --verbose flag. The deleted empty directories are simply ignored by node-glob-sync altogether: no console output appears at the time of deletion. The only time a directory is deleted is when it contains a file.

KeithGillette avatar Mar 31 '17 20:03 KeithGillette

@KeithGillette Okay, what a bummer. Thank you for instant feedback. The point is, I do use chokidar for file watching and paulmillr/chokidar#566 describes exactly your bug with deletion of empty directories. Unfortunately I can't reproduce this as long as I don't have a Windows 10 machine available.

AndyOGo avatar Mar 31 '17 20:03 AndyOGo

Bummer, indeed. My current needs don't actually require globbing functionality, so I have fallen back to node-sync-files which does not suffer from this issue. However, that leads me to believe that the problem may not be related to the open chokidar issue, as node-sync-files also has that as a dependency.

KeithGillette avatar Mar 31 '17 21:03 KeithGillette

Sure, use the tool which satisfies your needs best.

Hmh, interesting. But I don't think so because node-sync-files is using an outdated chokidar version ^1.0.0-rc3 and node-sync-glob is using a more up-to-date version ^1.6.1 of chokidar.

I will keep an eye on this and try to get access to a windows 10 box, so that I can prove what is going on exactly.

AndyOGo avatar Apr 01 '17 06:04 AndyOGo

Thanks again, @AndyOGo. While we can get by with a straight directory sync now, I foresee use cases for globbing in my project in the future, so I hope you are able to resolve the issue. If it would be helpful in testing, I can get you access to a virtualized Windows 10 environment on Azure.

KeithGillette avatar Apr 01 '17 10:04 KeithGillette

@KeithGillette Thanks for your offer to help. Meanwhile I asked my brother if he has windows 10, and if I can grap his notebook. So I will see what I can find out and post my results here as soon as I'm done.

AndyOGo avatar Apr 01 '17 21:04 AndyOGo

I just tested it on windows 10 and unfortunately I found even more bugs.

  1. Deleting empty sub directory does nothing, neither syncing nor error logging.
  2. Creating a New Folder does create a copy of New Folder, but after that folder has been renamed it does not sync it. - https://github.com/paulmillr/chokidar/issues/592
  3. Even worse If I rename a folder it does not delete the old one - https://github.com/paulmillr/chokidar/issues/593
  4. Each name of a rename will only sync at first time, later renaming with the same name do just nothing. - https://github.com/paulmillr/chokidar/issues/593
  5. Linux rename event - https://github.com/paulmillr/chokidar/issues/591

AndyOGo avatar Apr 02 '17 17:04 AndyOGo

I just installed node-inspector and it logs an error though the windows CLI does not print it...

Error watching file for changes: EPERM

I tracked the call stack down and chokidar is emitting this error

AndyOGo avatar Apr 02 '17 17:04 AndyOGo

I got a failing test now:

https://ci.appveyor.com/project/AndyOGo/node-sync-glob/build/1.0.71/job/mvjw85ufp2bhxxvg#L789

FAIL  test\sync.spec.js (8.87s)
  ● node-sync-glob watch › should sync empty sub directory deletion
    ENOTEMPTY: directory not empty, rmdir 'C:\projects\node-sync-glob\tmp\mock'
      
      at Object.fs.rmdirSync (fs.js:856:18)
      at rmkidsSync (node_modules\fs-extra\lib\remove\rimraf.js:300:11)
      at rmdirSync (node_modules\fs-extra\lib\remove\rimraf.js:289:7)
      at rimrafSync (node_modules\fs-extra\lib\remove\rimraf.js:254:7)
      at node_modules\fs-extra\lib\remove\rimraf.js:298:5
      at Array.forEach (native)
      at rmkidsSync (node_modules\fs-extra\lib\remove\rimraf.js:297:26)
      at rmdirSync (node_modules\fs-extra\lib\remove\rimraf.js:289:7)
      at Function.rimrafSync [as sync] (node_modules\fs-extra\lib\remove\rimraf.js:254:7)
      at Object.removeSync (node_modules\fs-extra\lib\remove\index.js:4:17)
  ● node-sync-glob watch › should sync empty sub directory deletion
    EPERM: operation not permitted, mkdir 'C:\projects\node-sync-glob\tmp\mock\bar\empty'
      
      at Object.fs.mkdirSync (fs.js:895:18)
      at Object.mkdirsSync (node_modules\fs-extra\lib\mkdirs\mkdirs-sync.js:29:9)
      at Object.ensureDirSync (test\helpers.js:13:154)
      at Object.<anonymous> (test\sync.spec.js:98:43)
  node-sync-glob watch

AndyOGo avatar Apr 11 '17 06:04 AndyOGo

Well the empty test dir tmp/mock/bar/empty is not recognized by glob-all on initial mirroring...

Windows: https://ci.appveyor.com/project/AndyOGo/node-sync-glob/build/1.0.75/job/yp5tk3x4sm246vj6#L865

console.log src\index.js:94
    [ 'tmp\\mock\\@org',
      'tmp\\mock\\@org\\a.txt',
      'tmp\\mock\\@org\\b.txt',
      'tmp\\mock\\@org\\d.txt',
      'tmp\\mock\\a.txt',
      'tmp\\mock\\b.txt',
      'tmp\\mock\\bar',
      'tmp\\mock\\bar\\c.txt',
      'tmp\\mock\\foo',
      'tmp\\mock\\foo\\b.txt',
      'tmp\\mock\\foo\\d.txt',
      'tmp\\mock\\transform.js' ]
 FAIL  test\sync.spec.js (8.917s)
  ● node-sync-glob watch › should sync empty sub directory deletion
    ENOTEMPTY: directory not empty, rmdir 'C:\projects\node-sync-glob\tmp\mock'
      
      at Object.fs.rmdirSync (fs.js:856:18)
      at rmkidsSync (node_modules\fs-extra\lib\remove\rimraf.js:300:11)
      at rmdirSync (node_modules\fs-extra\lib\remove\rimraf.js:289:7)
      at rimrafSync (node_modules\fs-extra\lib\remove\rimraf.js:254:7)
      at node_modules\fs-extra\lib\remove\rimraf.js:298:5
      at Array.forEach (native)
      at rmkidsSync (node_modules\fs-extra\lib\remove\rimraf.js:297:26)
      at rmdirSync (node_modules\fs-extra\lib\remove\rimraf.js:289:7)
      at Function.rimrafSync [as sync] (node_modules\fs-extra\lib\remove\rimraf.js:254:7)
      at Object.removeSync (node_modules\fs-extra\lib\remove\index.js:4:17)
  ● node-sync-glob watch › should sync empty sub directory deletion
    EPERM: operation not permitted, mkdir 'C:\projects\node-sync-glob\tmp\mock\bar\empty'
      
      at Object.fs.mkdirSync (fs.js:895:18)
      at Object.mkdirsSync (node_modules\fs-extra\lib\mkdirs\mkdirs-sync.js:29:9)
      at Object.ensureDirSync (test\helpers.js:13:154)
      at Object.<anonymous> (test\sync.spec.js:98:43)

In contrast OS X: https://travis-ci.org/AndyOGo/node-sync-glob/jobs/220861943#L1059

console.log src/index.js:94
    [ 'tmp/mock/@org',
      'tmp/mock/@org/a.txt',
      'tmp/mock/@org/b.txt',
      'tmp/mock/@org/d.txt',
      'tmp/mock/a.txt',
      'tmp/mock/b.txt',
      'tmp/mock/bar',
      'tmp/mock/bar/c.txt',
      'tmp/mock/bar/empty',
      'tmp/mock/foo',
      'tmp/mock/foo/b.txt',
      'tmp/mock/foo/d.txt',
      'tmp/mock/transform.js' ]

 PASS  test/sync.spec.js (9.997s)

  node-sync-glob watch
    ✓ should sync a file (2402ms)
    ✓ should sync an array of files (2397ms)
    ✓ should sync a directory (2429ms)
    ✓ should sync globstar (2428ms)
    ✓ should sync empty sub directory deletion (196ms)

AndyOGo avatar Apr 11 '17 08:04 AndyOGo

tmp/mock/bar folder does not exist on windows, hence no empty subfolder within can be created...

console.log src\index.js:101
    sources: tmp/mock/**/* -> tmp/mock/**/*
  console.log src\index.js:102
    target: tmp/sync -> tmp\sync
  console.log src\index.js:103
    globed files: 
    	tmp\mock\@org
    	tmp\mock\@org\a.txt
    	tmp\mock\@org\b.txt
    	tmp\mock\@org\d.txt
    	tmp\mock\a.txt
    	tmp\mock\b.txt
    	tmp\mock\bar
    	tmp\mock\bar\c.txt
    	tmp\mock\foo
    	tmp\mock\foo\b.txt
    	tmp\mock\foo\d.txt
    	tmp\mock\transform.js
  console.log src\index.js:237
    RAW: change -> b.txt
  console.log src\index.js:238
    { watchedPath: 'tmp\\mock\\foo' }
  console.log src\index.js:237
    RAW: change -> b.txt
  console.log src\index.js:238
    { watchedPath: 'tmp\\mock\\foo\\b.txt' }
  console.log src\index.js:237
    RAW: rename -> d.txt
  console.log src\index.js:238
    { watchedPath: 'tmp\\mock\\foo' }
  console.log src\index.js:237
    RAW: rename -> d.txt
  console.log src\index.js:238
    { watchedPath: 'tmp\\mock\\foo\\d.txt' }
  console.log src\index.js:237
    RAW: change -> foo
  console.log src\index.js:238
    { watchedPath: 'tmp/mock' }
 FAIL  test\sync.spec.js (8.937s)
  ● node-sync-glob watch › should sync empty sub directory deletion
    ENOTEMPTY: directory not empty, rmdir 'C:\projects\node-sync-glob\tmp\mock'
      
      at Object.fs.rmdirSync (fs.js:856:18)
      at rmkidsSync (node_modules\fs-extra\lib\remove\rimraf.js:292:11)
      at rmdirSync (node_modules\fs-extra\lib\remove\rimraf.js:283:7)
      at rimrafSync (node_modules\fs-extra\lib\remove\rimraf.js:252:7)
      at options.readdirSync.forEach.f (node_modules\fs-extra\lib\remove\rimraf.js:291:39)
      at Array.forEach (native)
      at rmkidsSync (node_modules\fs-extra\lib\remove\rimraf.js:291:26)
      at rmdirSync (node_modules\fs-extra\lib\remove\rimraf.js:283:7)
      at Function.rimrafSync [as sync] (node_modules\fs-extra\lib\remove\rimraf.js:252:7)
      at Object.removeSync (node_modules\fs-extra\lib\remove\index.js:6:17)
  ● node-sync-glob watch › should sync empty sub directory deletion
    EPERM: operation not permitted, mkdir 'C:\projects\node-sync-glob\tmp\mock\bar\empty'
      
      at Object.fs.mkdirSync (fs.js:895:18)
      at Object.mkdirsSync (node_modules\fs-extra\lib\mkdirs\mkdirs-sync.js:31:9)
      at Object.ensureDirSync (test\helpers.js:13:154)
      at Object.<anonymous> (test\sync.spec.js:100:45)
  node-sync-glob watch
    √ should sync a file (2190ms)
    √ should sync an array of files (2180ms)
    √ should sync a directory (2230ms)
    √ should sync globstar (2243ms)
    × should sync empty sub directory deletion
  console.log test\sync.spec.js:99
    EXISTS: tmp/mock/bar -> false
  console.log test\sync.spec.js:120
    { Error: EPERM: operation not permitted, mkdir 'C:\projects\node-sync-glob\tmp\mock\bar\empty'
        at Object.fs.mkdirSync (fs.js:895:18)
        at Object.mkdirsSync (C:\projects\node-sync-glob\node_modules\fs-extra\lib\mkdirs\mkdirs-sync.js:31:9)
        at Object.ensureDirSync (C:\projects\node-sync-glob\test\helpers.js:13:154)
        at Object.<anonymous> (C:\projects\node-sync-glob\test\sync.spec.js:100:45)
        at attemptAsync (C:\projects\node-sync-glob\node_modules\jest-jasmine2\vendor\jasmine-2.5.2.js:1984:24)
        at QueueRunner.run (C:\projects\node-sync-glob\node_modules\jest-jasmine2\vendor\jasmine-2.5.2.js:1939:9)
        at C:\projects\node-sync-glob\node_modules\jest-jasmine2\vendor\jasmine-2.5.2.js:1966:16
        at C:\projects\node-sync-glob\node_modules\jest-jasmine2\vendor\jasmine-2.5.2.js:1909:9
        at attemptAsync (C:\projects\node-sync-glob\node_modules\jest-jasmine2\vendor\jasmine-2.5.2.js:1987:9)
        at QueueRunner.run (C:\projects\node-sync-glob\node_modules\jest-jasmine2\vendor\jasmine-2.5.2.js:1939:9)
      errno: -4048,
      code: 'EPERM',
      syscall: 'mkdir',
      path: 'C:\\projects\\node-sync-glob\\tmp\\mock\\bar\\empty' }
 PASS  test\lib\sources-bases.spec.js

AndyOGo avatar Apr 11 '17 10:04 AndyOGo

A new file watcher just got released for node 😱 NSFW ... Node Sentinel File Watcher 🎉 I will try that

AndyOGo avatar Jul 09 '17 14:07 AndyOGo