Support for emitting "initialized" event after initial scan of watched directory

[flaw3d]

It would be nice to have an event emitted after the initial items are scanned. Since there is an ignoreInitial option, I assume this should be possible.

watcher.on ( 'initialized', () => {
  // All initial (add & addDir) events for pre-existing files & folders have just been emitted
});

If this is an enhancement that should be added, I would be happy to work on this if I can be pointed in the right direction.

[fabiospampinato]

What's the use case for this?

[aral]

@fabiospampinato In Kitten (which currently uses Chokidar), I listen for the ready event – which gets dispatched after the initial scan is complete – and set a flag which I use to decide whether or not to bubble events to the rest of the development server. The development server restarts itself when there is a change and I don’t want it to do that for every file that’s added in the initial scan. But I do need those events as I build by list of routes based on it as Kitten uses file-based routing. (Code snippet at end.)

Hope this provides a good enough use case :) (Please feel free to ask if I haven’t been able to explain it clearly.)

Also: when does the ready event in Watcher fire? (I would have assumed it would be when the initial file scan is complete.) (Found the answer: “No events are emitted before this event”) So yes, an initialised event would act like the ready event does in Chokidar, then.

For reference, in Chokidar:

.on('ready', () => log('Initial scan complete. Ready for changes'))

PS. Looking forward to seeing if I can replace Chokidar with Watcher for Kitten and getting rid of the native fsevents module. Thanks for making + sharing this. :)

this.watcher = chokidar
  .watch(watcherGlob, watcherOptions)
  .on('ready', async () => {
    this.initialised = true
    resolve(this.filesByExtensionCategoryType)
  })
  .on('add', (filePath, _stats) => { this.notifyListeners('file', 'added', filePath) })
  .on('change', filePath => { this.notifyListeners('file', 'changed', filePath) })
  // etc.

//…

async notifyListeners(itemType, eventType, itemPath) {
  if (this.initialised) {
    this.dispatchEvent(new CustomEvent('file', {
      detail: {
        eventType,
        itemType,
        itemPath  
      }
    }))
  }
}

[fabiospampinato]

Oh I see. I don't remember why I implemented the ready event that way 🤔 I suppose maybe it should be changed to work like in chokidar, or an additional event should be added.

[aral]

@fabiospampinato Hindsight is 20/20 and all that :)

Just did a quick test and Watcher works like a charm. Can’t believe they added recursive file watching for Linux in Node 19.1.0 and it’s still not implemented properly (it uses a 5-second polling interval as far as I can tell) *smh* :)

Assuming it would be a semver major update if you changed how the ready event works so it’s definitely your call. Do you value parity with Chokidar (for folks porting) or are you ok with a slightly different API? (initialized – I guess you’re using US spelling – would be a semver minor update). Either way, if you’d like me to prepare a pull request, just let me know; happy to.

[fabiospampinato]

Can’t believe they added recursive file watching for Linux in Node 19.1.0 and it’s still not implemented properly (it uses a 5 second polling interval as far as I can tell) smh :)

🤣 I guess I shouldn't use it then.

Do you value parity with Chokidar (for folks porting) or are you ok with a slightly different API?

I don't really care about Chokidar anymore, but the emitted events are close enough that alignment would be preferable if possible. I guess if there's no actual use case for the current "ready" event we might as well delete it, and use the name for Chokidar's version of the event. I'm not even sure if the current "ready" event is guaranteed the be emitted before any other non-error event 🤔

initialized – I guess you’re using US spelling – would be a semver minor update

My spelling is weak, I don't know if inited is grammatically correct 🤣, but initialized is a mouthful. Maybe we could go for init or something.

Either way, if you’d like me to prepare a pull request, just let me know; happy to.

I'm pretty busy with other stuff, but if the change is minor and the PR is easy to review that could accelerate things 👍

[aral]

@fabiospampinato Cool, I’d say let’s redefine what ready means, then, for parity with Chokidar. It’ll reduce the effort for folks coming over from Chokidar, which is never a bad thing :)

OK, let me look into this and prepare a PR. Should be simple enough (famous last words) ;)

[aral]

Famous last words, indeed :)

Having looked into it a bit, I’m not sure if this is possible given the current design. If you set ignoreInitial to false on a deep directory structure, only the first batch of events appears to have isInitial set properly.

e.g., I set the following watcher on the Watcher directory itself:

const watcher = new Watcher('.', { recursive: true, ignoreInitial: false })

And I also added the isInitial flag to the Event type so I could see if it was being set properly during the initial scan:

types.ts:

type Event = [TargetEvent, Path, Path?, Boolean?];

And passed that along in the eventsPopulate() method:

watcher_handler.ts:

  async eventsPopulate ( targetPaths: Path[], events: Event[] = [], isInitial: boolean = false ): Promise<Event[]> {
    await Promise.all ( targetPaths.map ( async targetPath => {
      const targetEvents = await this.watcher._poller.update ( targetPath, this.options.pollingTimeout );
      await Promise.all ( targetEvents.map ( async event => {
        events.push ([ event, targetPath, undefined, isInitial ]);
        // …
      }));
    }));
    return events;
  };

And then just logged out the events I was getting:

watcher_handler.ts:

  onTargetEvents ( events: Event[] ): void {
    for ( const event of events ) {
      console.log(event)
      this.onTargetEvent ( event );
    }
  }

In the initial scan, the first batch of results has isInitial correctly set to true. However, the second batch (still part of the initial scan) has it set to false:

…
[
  'addDir',
  '/var/home/aral/Projects/other/watcher/test/__TREES__/98/home/deep/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15',
  undefined,
  true
]
Add directory: /var/home/aral/Projects/other/watcher/test/__TREES__/98/home/deep/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15
[
  'addDir',
  '/var/home/aral/Projects/other/watcher/test/__TREES__/99/home/deep/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15',
  undefined,
  true
]
Add directory: /var/home/aral/Projects/other/watcher/test/__TREES__/99/home/deep/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15
[
  'addDir',
  '/var/home/aral/Projects/other/watcher/test/__TREES__/0/home/deep/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16',
  undefined,
  false
]
Add directory: /var/home/aral/Projects/other/watcher/test/__TREES__/0/home/deep/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16
[
  'addDir',
  '/var/home/aral/Projects/other/watcher/test/__TREES__/0/home/deep/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17',
  undefined,
  false
]
…

So this is likely going to require quite a refactor to get to work.

I don’t currently see a simple way of knowing when the initial scan is complete while listening for initial add events.

(And it’s very possible I’ve missed something simple or don’t entirely understand how it should work.)

[fabiospampinato]

It's a tricky library to work on, I'll take a look at this 👍

[fabiospampinato]

FWIW an alternative way of doing this would be to discover initial files and folders manually with tiny-readdir, then instantiating the watcher with the result of that (so that the scan isn't performance twice)

[aral]

It's a tricky library to work on, I'll take a look at this +1

Recursion always is :)

FWIW an alternative way of doing this would be to discover initial files and folders manually with tiny-readdir, then instantiating the watcher with the result of that (so that the scan isn't performance twice)

Haha, had just done a quick test with tiny-readdir when I saw your reply :) Good to know I can just pass the result of that in. (And yes, that should be good enough for what I need – thank you.) :)

[aral]

@fabiospampinato Quick update: I don’t think adding the readdirMap manually is working either (when you have one with depth > 1). Opened a separate issue for that here: https://github.com/fabiospampinato/watcher/issues/18

It’s not a blocker, just a performance issue, as it means the map has to be created twice.

[beorn]

Would also like to see this implemented. Was surprised when 'ready' was emitted so early on (after coming from Chokidar), and in any case need this kind of feature as I need to ensure I have the entire folder loaded before releasing control to the user. :-|

[kodelio]

+1 for this feature. Getting events for files already in the folder is quite useful

[beyeshooter]

any update ? how can we achieve this, thanks.