Old endpoints are not removed from Upstream (envoy cluster) when using a custom Upstream plugin

[mwieczorek]

Gloo Edge Version

1.12.x (latest stable)

Kubernetes Version

1.21.x

Describe the bug

I followed the tutorial to create a custom upstream plugin. (Originally, it was for AWS cloudmap, but to easily show the issue I created also a 'dummy' implementation)

The plugin returns always one endpoint for the upstream (of my new type), with IP equal to 10.0.0.X, where X = time.Now().Minute() - this weird logic is to simulate a change in endpoint returned by 'external source'.

Every minute, a different IP is returned, but in cluster's endpoints (glooctl proxy served-config) I still see the old endpoints.

Steps to reproduce the bug

Please take a look at my implementation available here: https://github.com/mwieczorek/gloo/tree/dummy (dummy branch, last commit has all the changes).

To recreate the issue:

• clone this repo
• git switch dummy # change branch to dummy
• kind create cluster
• make push-kind-images
• glooctl install gateway
• kubectl set image deployment/gloo gloo=quay.io/solo-io/gloo:0.0.0-fork -n gloo-system
• kubectl apply -f install/helm/gloo/crds/gloo.solo.io_v1_Upstream.yaml # update CRDs to have new upstream def
• kubectl create namespace dummy
• kubectl apply file like below:

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: dummy
  namespace: dummy
spec:
  virtualHost:
    domains:
      - dummy.example.com
    routes:
      - matchers:
          - prefix: /
        options:
          autoHostRewrite: true
          prefixRewrite: /
        routeAction:
          single:
            upstream:
              name: dummy
              namespace: dummy
---
apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: dummy
  namespace: dummy
spec:
  dummy:
    name: dummy

• check the cluster's endpoint like:

glooctl proxy served-config | grep "clusterName: dummy" -A 30

each minute, the list of endpoints gets a new item (the new one with 'current minute', but the old one is not removed). If I delete gloo pod, the list gets once again only one endpoint (but after a minute there will be 2, another minute 3, and so on...)

Expected Behavior

When my 'external source' for IPs for endpoints returns new values, only those should be set in the upstream/cluster endpoints list.

Additional Context

No response