Allow partial subscriptions to secrets

[onemanbucket]

trafficstars

https://github.com/envoyproxy/envoy/issues/21393

[mpuncel]

IMO there could in theory be users relying on this current behavior as it can be dangerous to remove a secret that Envoy still needs from a snapshot. I'd be okay with making this configurable, but I don't think it should be merged as-is.

I also asked some questions about your setup on https://github.com/envoyproxy/envoy/issues/21393 since (at first glance) it seems there might be a simpler way to set up your resource relationships to avoid this problem.

[onemanbucket]

dangerous to remove a secret that Envoy still needs from a snapshot

But according to the xDS spec[1], omitting a non cluster/listener resource from the ADS response does not mean that the client (envoy) should remove it. It should instead be retained until there are no internal references left (in our case, when the old listener is drained and removed). It seems like envoy is behaving correctly, but without this patch applied to the java control plane, the secrets subscription will fail since envoy subscribes to both new and old secret, while the snapshot only now contains the new one.

There might be some ambiguity in the spec in what the rules are for resource subscriptions?

~And yes, we can probably work around the problem, but unfortunately not without adding complexity to our setup.~ Thanks for suggesting to inline the secrets in the envoy issue, that unblocks us until this is merged!

[1] https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol#deleting-resources