New tail_mode and stat_poll_interval settings to support not keeping file handles permanently open

[david-garcia-garcia]

Fixes https://github.com/crowdsecurity/crowdsec/issues/4074

This PR adds two new configuration options:

• tail_mode: default or stat
• stat_poll_interval: how frequently should files be checked for changes when tail_mode=stat

To avoid introducing regresions, I built an abstraction to wrap the actual tail implementation depending on tail_mode. When using tail_mode the previous implementation used. When using tail_mode=stat, a new tail implementation that does not permamently keep the file handle open is used.

[github-actions[bot]]

@david-garcia-garcia: There are no 'kind' label on this PR. You need a 'kind' label to generate the release automatically.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind fix
• /kind chore
• /kind dependencies

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions[bot]]

@david-garcia-garcia: There are no area labels on this PR. You can add as many areas as you see fit.

• /area agent
• /area local-api
• /area cscli
• /area appsec
• /area security
• /area configuration

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[codecov[bot]]

Codecov Report

:x: Patch coverage is 83.23353% with 28 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 62.74%. Comparing base (09a1ed1) to head (3b12258). :warning: Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
.../acquisition/modules/file/tailwrapper/stat_tail.go	82.69%	13 Missing and 5 partials :warning:
...acquisition/modules/file/tailwrapper/nxadm_tail.go	82.85%	4 Missing and 2 partials :warning:
pkg/acquisition/modules/file/run.go	88.23%	2 Missing :warning:
...kg/acquisition/modules/file/tailwrapper/factory.go	81.81%	2 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4075      +/-   ##
==========================================
+ Coverage   62.62%   62.74%   +0.11%     
==========================================
  Files         456      459       +3     
  Lines       33076    33228     +152     
==========================================
+ Hits        20715    20849     +134     
- Misses      10240    10253      +13     
- Partials     2121     2126       +5     

Flag	Coverage Δ
bats	46.19% <30.53%> (-0.12%)	:arrow_down:
unit-linux	35.54% <82.63%> (+0.26%)	:arrow_up:
unit-windows	24.84% <59.88%> (+0.15%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[david-garcia-garcia]

/kind enhancement /area agent

[mmetc]

Hi @david-garcia-garcia,

Thanks for the PR!

We're close to releasing 1.7.4, so this won't be merged before 1.7.5.

As you can see from the last few commits in the master branch, we are in the middle of a global refactor of the acquisition: the current non-blocking StreamingAcquisition() method is being replaced by a synchronous Stream() method. This lets us drop tombs in favor of context-based cancellation (more errgroup, waitgroup and fewer channels), and bubble all retryable errors up to a centralized backoff loop.

For reference, the journalctl datasource is already migrated and merged; syslog is migrated but not merged yet. Docker and file are likely to be next, shortly after the 1.7.4 release.

Since your PR is a wrapper around the tail package there shouldn't be a lot of conflicts if we postpone the review and adapt it to the new interface and concurrency model once they are in place.

Thanks again for your work and your patience!