Test and verify that Fluentd and log-manager behaviour is consistent on Azure

[aarnq]

Description

The Azure integration for audit and sc-logs has been under a few iterations and before promoting the feature further we would like to do proper testing to ensure that it works as expected.

So the goal is to first ensure that its behaviour is correct, that we actually get audit and sc-logs from Fluentd, second ensure that log-manager compaction can compact new logs into per day chunks, and third ensure that log-manager retention can remove old logs that is outside of the retention window.

Additional context

Refer to the behaviour of the S3 implementation as that has been proven to work and runs at scale.

Definition of done

• [ ] Azure implementations of audit and sc-logs are correct and consistent with the S3 equivalents.

[vomba]

I had a look at this, one of the problems is when using an ephemeralVolume log-manager seems to fail to create the folder to work in scratch/lm. Using emptyDir{} seems to work, i suggest we change it that for Azure it uses that.

[aarnq]

I had a look at this, one of the problems is when using an ephemeralVolume log-manager seems to fail to create the folder to work in scratch/lm. Using emptyDir{} seems to work, i suggest we change it that for Azure it uses that.

That sounds exactly like a bug to fix to me, rather than to accept that as a limitation that make it hard to use with large log volumes and small node volumes.

[vomba]

Will look more into it then.

[raviranjanelastisys]

Fluentd and log-manager Testing is being tested here

Fluentd - https://github.com/elastisys/compliantkubernetes-apps/pull/2426 Log-Manager - https://github.com/elastisys/compliantkubernetes-apps/pull/2428

[Pavan-Gunda]

Have we verified that it works after the upgrade @vomba ?

[vomba]

I have tested both and they are working great, i would still suggest that someone else does the testing and have a second look to confirm.

[raviranjanelastisys]

I think we can close this and we will get to know in the next app release and if needed we can reopen.

[aarnq]

I have verified with a running environment that this works as expected.