Keep active references to snapshots delivered

[nicoche]

Hey!

First, thanks for the work on go-control-plane.

I have a feature request. The idea would be to be able to know, at any time, what Resources are in use for each Envoy node connected to the XDS server.

Currently, the SimpleCache holds, for each connected node, what Snapshot should be delivered. afaiu, this is not sufficient to know which resources are acknowledged by a node: a snapshot can be in cache but not delivered yet or the target node could be rejecting the snapshot. Being able to introspect the actual configuration of Envoy nodes is pretty useful to implement strong guarantees on the control plane.

It seems possible to implement this feature in e.g. SimpleCache:

• Whenever a snapshot is set in cache using e.g. SetSnapshot, save it to a "fresh snapshots" list
• Whenever a version is acknowledged by a node, save this information (already done afaiu)
• Implement a method GetInUseSnapshot(nodeId string) *Snapshot which would join the fresh snapshots list and the node's in use version

The downside is obviously higher memory usage. This could be an opt-in feature.

What do you think? Is it something you would want to see integrated in go-control-plane? I'm writing a POC for this, I can let you know how it runs when it's ready.