File acq defaults in 1.5 issues

[LaurenceJJones]

Just to make it clearer file systems that are network shared or NFS do not work with inotify by default unless you have installed an optional package for the operating system. NOTE this does not resolve in all instances and following the workaround below is the most optimal solution.

This issue is intended to explore whether CrowdSec can detect the filesystem and change the default behaviour accordingly.

What would you like to be added?

By default on 1.5 file acquisition uses inotify instead of polling ( < 1.5 defaults). Many users are reporting that upgrading to 1.5 causes there to be no logs parsed since inotify does not work it does not inform the tailing acquisition to parse the lines.

Workaround:

User can enable the old defaults by adding poll_without_inotify: true to the file acquisition within acquis.yaml example below.

poll_without_inotify: true
source: file
filenames:
 - /tmp/foo/*.log
 - /var/log/syslog
labels:
 type: syslog

/kind enhancement

Why is this needed?

If crowdsec can detect that inotify failed to bind or there was some sort of error the system could auto heal from this instead of the user finding the option to turn back to manual polling

[github-actions[bot]]

@LaurenceJJones: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions[bot]]

@LaurenceJJones: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind bug
• /kind packaging

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[LaurenceJJones]

I have asked affected user to post there current workarounds or suggestions here

[GNU-Plus-Windows-User]

at least for me ionotify is working fine, yet disabling it only mostly fixes the issue.

Establishing watches...
Total of 1 watches.
Finished establishing watches, now collecting statistics.
Will listen for events for 30 seconds.
total  access  modify  close_write  close_nowrite  open  filename
48     6       14      7            7              14    t

[yuphing-ong]

1. definitely should highlight this in documentation
2. ~~perhaps as part of agent startup, you can do some testing e.g. create a temp file, start inotify on it, modify said file, then after 1 second check inotify, and if no events generate a warning.~~
3. if (2) above is doable, then you can also decide whether auto-heal is an option, and generate the warning that inotify is not detecting changes, so defaulting to polling instead.

OK ignore (2), I think the way NFS works the system doing the changes will detect the changes, but changes are not propagated to the network till later. So it will not be possible to detect this problem on the crowdsec host.

Another possibility is to explicitly test using os.Stat and fileInfo.Sys the Fstype and warn if it's not ext2/3/4 or xfs? It won't help on the main server (since it's not an NFS mount), but the distributed server agents will get the warning.

[mateuszdrab]

I've experienced an issue since this setting default kicked in - inotify seems not to work for me in K3s on Ubuntu so had to revert back to poll_without_inotify: true otherwise logs would not get ingested after some time. The files are locally on the filesystem so not the SMB/NFS clause does not apply here - not sure what is wrong. force_inotify: true is already true which is the setting I initially suspected would be responsible for crowdsec missing new log files but perhaps I was wrong.

[LaurenceJJones]

#2241

[mateuszdrab]

#2241

Don't think this PR would make any difference for me, my log files are on the local filesystem.

[MrAlucardDante]

/kind enhancement