linkerd2
linkerd2 copied to clipboard
linkerd
Monitoring outbound HTTPS external call
Feature Request
I would like Linkerd2 to monitor external service calls using HTTPS protocol. I would like to get at least the outbound info, the rps and the error rates.
What problem are you trying to solve?
We are deploying our services under Kubernetes / EKS on AWS. We are using many AWS services and most of them are using HTTPS (SQS, SNS, etc...). A common pattern is the following:
Service 1 ------> SQS -----> Service 2 -----> Service 3 (inside EKS) -----> Service 4 (outside EKS)
Only the communication between the Service 2 and the Service 3 will be monitored because they are running inside Kubernetes.
The communication between Service 2 and the Service 4 will not monitored because the communication uses HTTPS.
How should the problem be solved?
N/A
What do you want to happen? Add any considered drawbacks. N/A
Any alternatives you've considered?
The Service1 could create a compatible metric to monitor outbound call The Service 2 could create a compatible metric to monitor inbound call
The issue is related to grafana/prometheus because most of the graphics are bound to a deployment / pod and the external service doesn't have it.
Is there another way to solve this problem that isn't as good a solution?
How would users interact with this feature?
If you can, explain how users will be able to use this. Maybe some sample CLI output?
Another solution, could be to offer a rest api to interact with the sidecar. The api could offer the capability to
- increment outbound counter (request_total, response_total)
- send the response_latency timer
This is a great use case. There will need to be tradeoffs, as the proxy will have to MITM the data. There are two possible options that I can think of:
- Add a linkerd cert to the trust chain locally (so that it is accepted for everything), terminate in the proxy and verify on the outbound hop. This'll require container modifications or code changes.
- Configure apps to use
httpand configurehttpsurls that the proxy will upgrade to.
@jmirc any ideas whether the tradeoffs are worth it for either option?
@grampelberg I will prefer using the first solution even if it is more work for the first integration, it will be transparent for the client if we are using or not the proxy.
We're also looking to monitor a ~dozen external service calls, all over https. I had tried to get this working with linkerd for a couple hours back in December, not realizing that TLS was likely my problem.
Is it possible for linkerd to mitm outgoing https connections in such a way that per-endpoint config (besides trusting linkerd's cert) isn't necessary? I think trusting a cert/CA is probably an ok compromise for our purposes. If linkerd's control plane generates its cert as a secret, it can probably be mounted into the app's container in the right place for validation to succeed. Or we can statically generate a cert and stick it in our CI build somewhere.
In our case, it'd also be possible for us to specify a different url for our integrations if necessary. Every one of the libraries we use (so far) allow for a custom urls to be provided. It'd just be less friendly that way, since we'd have to maintain a fair amount of config. If that's what we do, I don't know that I particularly care about TLS between the app container and the sidecar linkerd.
Regardless of what happens, it'd be extremely important to me that linkerd would verify https certs for the connections it makes. I'd like failures to be fatal (like send a 500 back to my app) since I really don't want to be accidentally making insecure connections across the public internet.
@bdimcheff that's really useful. Sounds like it is worth looking into the MITM config a little bit more.
It appears that they require configuration. My understanding is that istio's egress is opt-in (eg all traffic is blocked by default), so that makes some sense.
A little off topic, but it looks like they are moving to using their own CNI: https://github.com/istio/cni
For application pods in the Istio service mesh, all traffic to/from the pods needs to go through the sidecar proxies (istio-proxy containers). This istio-cni CNI plugin will set up the pods' networking to fulfill this requirement in place of the current Istio injected pod initContainers istio-init approach.
This is currently accomplished (for IPv4) via configuring the iptables rules in the netns for the pods.
The CNI handling the netns setup replaces the current Istio approach using a NET_ADMIN privileged initContainers container, istio-init, injected in the pods along with istio-proxy sidecars. This removes the need for a privileged, NET_ADMIN container in the Istio users' application pods.
This is similar to the linkerd-cni approach.
yeah i think what's interesting about istio's docs above is that (in their example) it allows you to exec into the pod and then curl https://www.google.com, presumably without getting a cert error. That config must somehow configure a mitm, no?
fwiw it'd definitely meet our needs if we needed to define service profiles (or some other k8s crd config) for each external TLS service that we wanted linkerd to intercept. I'd like to avoid having to manually alter a bunch of config on our service itself, but linkerd config is totally reasonable/expected.
@bdimcheff as I understand the docs a curl https://www.google.com without a VirtualService will not be mitm'd or have any stats. I tried to figure out the egress prometheus metrics and gave up though, so I might be entirely wrong.
I still really like the idea of mitm, especially if the injector can put trust bundles in the correct place automatically. We'll just need to figure out how feasible it is. This crazy amount of config frightens me.
as I understand the docs a
curl https://www.google.comwithout a VirtualService will not be mitm'd or have any stats.
yeah that's my understanding as well, which I think is fine/expected. I appreciate the ways in which linkerd just magically works, but in this case I'm definitely willing to write some config. There may be some outgoing TLS connections that I legitimately don't want linkerd to intercept, and it seems reasonable to me for TLS MITM to be opt-in.
We have a use case where there are a bunch of 3rd party integrations we connect to and would love to tap, monitor, and metric this traffic using linkerd.
Really need this feature to monitor the external traffic from the pod for observibility.
