envoy
envoy copied to clipboard
envoyproxy
envoy doesn't request oauth2 secrets via ads or sds
Description:
We're dynamically configuring envoy via a control plane (ads). Sometimes the configuration has the OAuth2 filter enable, sometimes it doesn't. When envoy pulls down it's first configuration and it does have OAuth2 configured, everything works as expected. However, when we start with a configuration that doesn't have OAuth2 and change it to one that does, the token and hmac secrets are not requested.
Repro steps:
Envoy 1.22.2, Go 1.18 for the stripped down control plane, and sample upstream
openssl req -nodes -new -x509 -keyout upstream.key -out upstream.crt envoy -c envoy.yaml --component-log-level config:trace go run control-plane.go go run upstream.go
After running all the commands, you can test the upstream directly or via envoy by running:
curl -kv https://localhost:8443 curl -kv https://localhost:443
However, the upstream is irrelevant for this issue. You can change the configuration from no-oauth2 to oauth2 by pressing enter on the control-plane console. You will notice that envoy does get the new listener, with the OAuth2 filter and the references to the token and hmac secrets, however, they're never requested by envoy. I've also tried changing SDS to use ADS without luck.
Admin and Stats Output:
The stats endpoint on port 9001 will show when the OAuth2 credentials are not initialized. The oauth2 flow will fail.
cluster_manager.cds.version_text: "oauth2" listener_manager.lds.version_text: "oauth2" sds.cluster_validation_context.version_text: "oauth2" sds.listener_certificate.version_text: "oauth2" sds.listener_validation_context.version_text: "oauth2" sds.oauth2_hmac_secret.version_text: "" sds.oauth2_token_secret.version_text: ""
Config:
Please review attached envoy.yaml
Logs:
We start the control plane with the configuration without oauth2. The logs confirm that it started successfully.
setting snapshot to index 0 version no-oauth2
Please review attached envoy-no-oauth2.log
After we press enter on the control plane, the oauth2 configuration gets set and envoy should update via xds, however, no auth2 secrets are requested.
setting snapshot to index 1 version oauth2
Please review attached envoy-oauth2.log
Code:
Please review attached upstream.go and control-plane.go
@marcosrmendezthd and @kyessenov, do you recall if this was working at all at some point? If so, at what version?
In our tests we find that the behaviour is random in some sense. At times we see SecretDiscoveryServiceServer.StreamSecrets being called and at other points we don't see it being called at all.
Please also see this issue we created in the go-control-plane: https://github.com/envoyproxy/go-control-plane/issues/581.
@mbana so it works the first time envoy gets a config. I also don't believe it's a control plane issue. I have tested with a stripped control plane as well, envoy is just not asking for the secrets. if you restart envoy, after seeing that secrets haven't been initialized, it works. I haven't seen other generic secrets being used (at least in the filters/configs that we use), maybe we're missing something for them on envoy? oh, and to answer your question, I'm not sure if this issue is on other envoy versions as well.
@marcosrmendezthd, @kyessenov, @adisuissa,
Do you know who to reach out to about this issue and potentially getting it fixed.
On my side, I can collect logs etc.
I wanted to ask, how do I enable verbose gRPC logging. I want to see logging when a connection to SDS is established, and when it is not established as in my case, I see random behaviour.
The issue is likely something to do with the initialization order of SDS and filters, not transport level with gRPC. Someone would need to figure out the race condition and propose a fix. If anyone finds time or is in critical need, they should be able to figure this out.
The issue is likely something to do with the initialization order of SDS and filters, not transport level with gRPC. Someone would need to figure out the race condition and propose a fix. If anyone finds time or is in critical need, they should be able to figure this out.
@kyessenov, any tips on hints on where to start? I haven't done C++ in around 10 years.
@mbana @marcosrmendezthd If possible, can you please try https://github.com/envoyproxy/envoy/pull/23356?
@mbana @marcosrmendezthd If possible, can you please try #23356?
what's the official dev docker image I can use to test it @mbana ? 😄