elastic
Incremental Sync should consider updates that doesn't change timestamp
Problem Description
Incremental Sync in various connectors are not updating files when parent folder permissions are changed.
Our incremental sync logic is quite naive to detect updates, we check timestamps for figuring updated files. If there is a change that affects the file but doesn't change the timestamp, we are simply ignoring this files for incremental sync.
An example is when Permissions for the Parent folder permissions changed in Google Drive connector, incremental sync doesn't update sub files as timestamps are not changed.
Proposed Solution
While it could be security/permissions related issue, we should consider checking permission updates and mark all the files affected to be updated.
(Disclaimer it is not discovered throughly, maybe a better way to do that) However for Google Drive, I don't think we have a good way to do it though as figuring out what permission is changed or what is affected is not an easy task. That may require storing some kind of information on permissions to figure out changes.
Alternatives
Another option is to figure out a common way to store permission/file information and update the files either on framework level or per connector that is needed.
