Envoy Proxy deployment is not cleaned up when we customize the deployment name, replicas using EnvoyProxy resource

[tejaboppana]

Description: Envoy Proxy deployment is not cleaned up when we customize the deployment name, replicas etc using EnvoyProxy resource,

Repro steps:

1. Create a Gateway and use a EnvoyProxy to set a custom name and increase the replicas.
2. I tried deleting the Gateway and expected the envoy gateway to delete the envoy proxy deployment and service, but it did not. '
3. Next I created another Gateway with an EnvoyProxy resource. Then I deleted the EnvoyProxy, envoy gateway created a single replica deployment with the a default name but did not delete the older deployment.
4. Now when I deleted the Gateway , the new deployment with default name and the service were deleted but the older deployment with custom name still exists.

Environment: Envoy Gateway version - 1.1.2

Logs: I could not see any logs in Envoy Gateway pods when I deleted the Gateway.

[arkodg]

@tejaboppana thanks for flagging this in step 1, did you create the Gateway resource first (a.) and then create a EnvoyProxy specifying the custom deployment name (b.) and then link the EnvoyProxy resource to the Gateway by editing it (c.) ?

For this case the I think the infra layer will create a new deployment with the new name but not delete the older deployment When a custom name is set, we also need to delete the older deployment https://github.com/envoyproxy/gateway/blob/0f4cb27f0110051f7811122a2dd73652c93a15b2/internal/infrastructure/kubernetes/proxy/resource_provider.go#L153

[tejaboppana]

@arkodg I tried 2 things and below are my observations

1. Steps that you mentioned i.e a) Create Gateway b) Create EnvoyProxy c) Edit Gateway to link it with EnvoyProxy In this case a new deployment is created but the older deployment is not deleted which is not ideal. In addition to that, when I delete the Gateway and EnvoyProxy, I expected everything to be cleaned up. Only the older deployment and the service are deleted but the deployment with custom name is not deleted.

2. a) Create EnvoyProxy b) Create Gateway and link it to the EnvoyProxy In this case the deployment is created with the custom name that I specified which is expected. But when I delete the Gateway and EnvoyProxy resources I expected the deployment and service to be deleted. However, this did not happen and both the deployment and service were still running. I had to manually delete them.

[arkodg]

thanks, yeah an additional item here to delete based on gateway owning labels, and not just deployment to svc name https://github.com/envoyproxy/gateway/blob/0f4cb27f0110051f7811122a2dd73652c93a15b2/internal/infrastructure/kubernetes/infra_resource.go#L402

[zhaohuabing]

@zirain can we consider this as done? Can it be closed?

[arkodg]

@zhaohuabing there's additional work here of cleaning up the resource when the name has changed

[zirain]

@tejaboppana can you share a reproducable yaml configuration?

[arkodg]

@zirain the issue here when the name changes, https://github.com/envoyproxy/gateway/blob/82ce107e76bfc9b8d7f1e2704c7d294bbb8b9606/internal/infrastructure/kubernetes/proxy/resource_provider.go#L322 we need to also delete the previously named resource, but this is called during the IR create/update flow

[zirain]

I think deletion is fine, because we use labels to filter now. Still need to a patch for update.

Step 1:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg-custom-name
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: All
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-name
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        name: custom-name

Step 2:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg-custom-name
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: All
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: custom-name
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-name
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        name: custom-name

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[kahirokunn]

keep

[showchan33]

I created k8s resources using only the manifest based on Step 2 of the link below. https://github.com/envoyproxy/gateway/issues/4380#issuecomment-2449326778 At this time, the Step 1 manifest is not APPLIED in advance.

Then, in addition to custom-name, a Deployment named envoy-default-envoy-gateway-12b6bb46 was also created. As a side note, the resource name of the Gateway I created is envoy-gateway, which is different from the eg-custom-name in the sample above.

$ kubectl get deploy -n envoy-gateway-system
NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
custom-name                            1/1     1            1           65s
envoy-default-envoy-gateway-12b6bb46   1/1     1            1           65s
envoy-gateway                          1/1     1            1           45h

On the other hand, when the Gateway was created after the EnvoyProxy was created, only the custom-name Deployment was created and not the envoy-default-envoy-gateway-12b6bb46.

[illrill]

I've noticed this behaviour applies not only for the Deployment but also for the Service.

[zirain]

@illrill the behaviour is same for all resources supported changing name.

[travisghansen]

Yes, deployments, hpa, svc, etc