Collect kubernetes events

[lllamnyp]

To let Cozystack administrators diagnose issues with the platform, kubernetes events, just like logs, need to be collected into some long-term storage.

At a minimum each event, represented as a JSON object, should be sent to VictoriaLogs. A pod running

kubectl get events --all-namespaces --watch --output=json

would almost do the trick, though there are some caveats with formatting the output (e.g. there are no newlines between the events), deduplicating events sent to vlogs, stripping irrelevant pod metadata, etc.

An improvement might be a small app talking directly to kubernetes' events API that would enable arbitrary output formatting. Better yet, it could send events directly to vlogs, without stdout as an intermediate step. Some of this is probably available as open-source tooling.

Finally, a more comprehensive solution, such as Kspan could be considered.

Further reading: How to collect Kubernetes events