watchdog icon indicating copy to clipboard operation
watchdog copied to clipboard

inotify recursive scan is not supported with event_filter that does not include IN_CREATE

Open MPalarya opened this issue 2 years ago • 1 comments

We're using the latest master which is 3.0.0 release with the addition of https://github.com/gorakhargosh/watchdog/commit/48c49a1fe0538be455d52aa9a0eb57519a6ce2eb

We're only interested in move and delete events, specifically of files, but we'd like to watch for files inside subdirs as well, so we're defining it like this:

observer.schedule(
    event_handler=our_handler,
    path=our_path,
    recursive=True,
    event_filter=[FileMovedEvent, FileDeletedEvent],
)

I understand that inotify does not support recursive scanning, and based on the implementation here:

        def _recursive_simulate(src_path):
            events = []
            for root, dirnames, filenames in os.walk(src_path):
                for dirname in dirnames:
                    try:
                        full_path = os.path.join(root, dirname)
                        wd_dir = self._add_watch(full_path, self._event_mask)
                        e = InotifyEvent(
                            wd_dir,
                            InotifyConstants.IN_CREATE | InotifyConstants.IN_ISDIR,
                            0,
                            dirname,
                            full_path,
                        )
                        events.append(e)
                    except OSError:
                        pass
                for filename in filenames:
                    full_path = os.path.join(root, filename)
                    wd_parent_dir = self._wd_for_path[os.path.dirname(full_path)]
                    e = InotifyEvent(
                        wd_parent_dir,
                        InotifyConstants.IN_CREATE,
                        0,
                        filename,
                        full_path,
                    )
                    events.append(e)
            return events

the approach seems to be to add the WDs when either a file is CREATEd or a directory fires its own event (create, modify, whatever)

so in order to achieve what we're looking for, we need to add DirCreatedEvent in addition to the two events that we're after. which means more CPU and RAM, slower performance and noisy debug logs

how can this be improved? I would expect that when selecting recursive=True internally, we will implicitly add IN_ISDIR without IN_CREATE to reduce the noise of all file creations, and still identify new directories created to add watches

MPalarya avatar May 22 '23 16:05 MPalarya

cc @marioga & @BoboTiG

MPalarya avatar May 30 '23 07:05 MPalarya