File connector as a scheduled job

[skprasadu]

The current file connector has ability where we zip a bunch of file and upload them and it gets indexed and then we can start searching for them.

But Is it possible to build a file connector that implements PollConnector, where given a network file path, will periodically poll new files getting added and index them and we can search them.

Are there any possibility of implementing this. If not I can attempt to implement something like this following this instructions @ https://github.com/danswer-ai/danswer/blob/main/backend/danswer/connectors/README.md . Will it be of value?

Thanks Krishna

[eojthebrave]

I think a file connector that would either poll a directory for new files, and maybe also re-ingest a file if its updated date changes, would be super helpful. As it is if your files change you need to delete the existing connector, create a new one, and then re-index the files.