Discover exact K8s Service from http_events

[rosenhouse]

Problem

A given Kubernetes pod may be the backend (target) for multiple Kubernetes services. I'd like to be able to determine, from the http_events table, the exact Kubernetes service that a request was directed to, or to detect that the request was sent directly to a pod IP instead of a service.

As far as I understand, today there isn't a reliable way to do this.

Current (unreliable) approach

Right now the best approach seems to be:

df.this_pod_id = df.ctx['pod_id']
df.is_server_tracing = df.trace_role == 2
df.dest_service_names = px.select(df.is_server_tracing, px.pod_id_to_service_name(df.this_pod_id), px.service_id_to_service_name(px.ip_to_service_id(df.remote_addr)))

In the common case where the client and server are both in-cluster and monitored by Vizier, then the record in http_events has trace_role == 2 and the record is from the perspective of the server pod.

If there is more than one service targeting that pod, thenpx.pod_id_to_service_name(df.this_pod_id) will return a list of service names.

One can then compare the contents of the Host field in req_headers against the elements in the dest_service_ids list.

This is unreliable, since http clients may not set the Host header to the name of the Kubernetes Service. This can occur, e.g. if the client application caches DNS results or uses other service-discovery mechanisms. Without relying on the Host header, if there are multiple services targeting the server pod, then it isn't possible to tell which one the client connected to. It also isn't possible to tell if the client bypassed the service(s) and connected directly to the pod IP.

Describe the solution you'd like

Ideally, the http_events record would include the destination IP and port that the client used when connecting to the server. From that, we could call ip_to_service_id.

When the server is outside the cluster and Pixie sees the client, then http_events includes that destination address information in remote_addr and remote_port. But when the server is traced, those fields are from the "server's perspective" and thus describe the apparent client address instead.

Describe alternatives you've considered

Besides the unreliable approach I describe above, perhaps there's some way to separately monitor TCP connect events from the client-side, grab the destination address from there, and merge that data with the http_events data, though I don't know exactly how.