Investigate how to send Envoy access logs to external data store like a SEIM

[phillipgibson]

Please describe the Improvement and/or Feature Request

Scope (please mark with X where applicable)

• New Functionality [X]
• Install [ ]
• SMI Traffic Access Policy [ ]
• SMI Traffic Specs Policy [ ]
• SMI Traffic Split Policy [ ]
• Permissive Traffic Policy [ ]
• Ingress [ ]
• Egress [ ]
• Envoy Control Plane [X]
• CLI Tool [ ]
• Metrics [ ]
• Certificate Management [ ]
• Sidecar Injection [ ]
• Logging [ ]
• Debugging [ ]
• Tests [ ]
• CI System [ ]
• Demo [ ]
• Project Release [ ]

Possible use cases Currently Envoy access logs can be viewed on a per pod basis using the command "kubectl logs -n -c envoy [options]". The more useful need for collecting Envoy access logs is to have to ability to correlate traffic events and have long-term availability to the logs for possible forensic analysis.

User Story - As a SecOps user, I would like the ability to configure OSM to send Envoy access logs in the mesh to an external data store (SEIM), so I can replay traffic events in needed for further analysis.

[keithmattix]

We should use this chance to leverage OpenTelemetry support: https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/access_loggers/open_telemetry/v3/logs_service.proto

[shashankram]

Previously, we used FluentBit for this. You can run it as a sidecar to stream logs to an external data store. The FluentBit configuration can be tweaked to parse and forward access logs.

[keithmattix]

Yeah, the advantage of the OTel approach is that it's not specific to fluentbit and it occurs directly in the Envoy proxy, so you prevent the Yet-Another-Sidecar problem

[shashankram]

Yeah, the advantage of the OTel approach is that it's not specific to fluentbit and it occurs directly in the Envoy proxy, so you prevent the Yet-Another-Sidecar problem

That does seem like a more integrated solution with Envoy that would be preferable.

[phillipgibson]

I'm in agreement with using OTel to make this happen.

[github-actions[bot]]

Added default label size/needed. Please consider re-labeling this issue appropriately.