beyla
beyla copied to clipboard
grafana
Beyla showing traffic that does not exist
The metric affected seems to be http_client metrics, and as far as I've been able to see so far it seems to be primarily clients written in c# that produces these faulty metrics.
Here's an example screenshot from prometheus that shows erroneous data with traffic that does not exist.
This service communicates with filestore which is a google product in GCP. But it shows as if the services is communicating with beyla.
Running beyla 2.0.4 deployed via helm Beyla running as a daemonset in Google kubernetes engine - standard cluster
@bjor-joh I assume here the 'redacted-service' should be your service that is communicating with the storage backend in GCP. So the metric is probably correct, just with server_address showing Beyla rather than the storage service.
server_address normally contains the name of the service called, but I am unsure of the behavior if this is something outside of the Kubernetes cluster. @marctc I believe it is supposed to show the hostname?
@bjor-joh I assume here the 'redacted-service' should be your service that is communicating with the storage backend in GCP. So the metric is probably correct, just with server_address showing Beyla rather than the storage service.
server_address normally contains the name of the service called, but I am unsure of the behavior if this is something outside of the Kubernetes cluster. @marctc I believe it is supposed to show the hostname?
@marevers yes exactly. Please note that this is not limited only to services outside the cluster. We also see a lot of in cluster communication that has the same issue.
So http_client_metrics for service-A show that it's making calls to service-B, while in reality no such traffic exists. It's more or less the same services that exhibit this behaviour, all of them are written in c#.
@marevers have you had any chance to look at this? Or can you tell me if there are any plans to look at it?
This is causing quite some issues for us since it's hard to figure out the fake traffic from the real traffic. Some of our services looks like they have 80+ down stream dependencies that actually does not exist
Hey @bjor-joh, thanks for the reminder and related with another issue. Will look into it soon!
