Support not keeping file handles permanently open when using file acquisition

[david-garcia-garcia]

What would you like to be added?

/kind feature

Add configuration options to allow file acquisition to work without permanently keeping file handles open.

Why is this needed?

The motivation is to support tailing files that have a SMB/SAMBA underlying storage. SMB/SAMBA implementation will prevent tailed files from being deleted if a file handle is open.

A special flag can be used to avoid this FILE_SHARE_DELETE but it is only available on Windows, plus it won't be available if SAMBA/SMB is used underneath an abstraction layer like Kubernetes CSI.

https://learn.microsoft.com/en-us/rest/api/storageservices/managing-file-locks

If this flag [FILE_SHARE_DELETE] isn't specified, any request to delete the file will fail, until the file is closed.

https://www.samba.org/samba/docs/4.5/man-html/smb.conf.5.html

When set to yes (default): Samba checks file system permissions directly and denies deletion if permissions don't allow it This aligns with Windows semantics where deletion permissions are verified at the time of the delete request

[github-actions[bot]]

@david-garcia-garcia: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions[bot]]

@david-garcia-garcia: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind bug
• /kind packaging

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[blotus]

Hello,

If I understood correctly, I don't think it's possible.

By default (on linux), we are using inotify to be notified of changes on a file. The notification does not include the changes, just that something was written to the file, so we have a keep a FD to be able to get the new content (in theory, it would be possible to keep opening/closing the file, but it would be extremely heavy if the file gets written to at any moderate speed, and it would just make it kinda random if you can actually delete the file).

If inotify cannot be used (for example, the file is a symlink or is in a network file system such as NFS or SMB), we revert to a basic polling method which stats the file to see if it changed size. Again, we need to keep a FD to be able to read the content of the file. In this situation, opening/closing the file each time could be a little bit more acceptable (as it wouldn't depend on the amount of writes it receives), but I don't really like it.

Regarding FILE_SHARE_DELETE, this could indeed be used but here the issue lies with go: it's not possible to pass this flag when using the standard library to open a file, and would require the use of windows lower-level APIs to read the files, which I'd rather avoid.

[david-garcia-garcia]

@blotus totally agree with what you say.

Again, we need to keep a FD to be able to read the content of the file. In this situation, opening/closing the file each time could be a little bit more acceptable (as it wouldn't depend on the amount of writes it receives), but I don't really like it.

I know that opening and closing the file is not the most elegant or performant solution, but is the only one. FILE_SHARE_DELETE is not available on POSIX. Also the underlying SMB/SAMBA might be wrapped with something else (like a volume mount in kubernetes with CSI) where those flags won't work.

By not keeping a FD you loose the ability to properly deal with file rotation (move) which in any case is not reliable on SAMBA/SMB.

I made a proposal here: https://github.com/crowdsecurity/crowdsec/pull/4075