Run Envoy Gateway like DaemonSet

[zvlb]

Description: For many use-cases need to start Kubernetes API Gateway like DaemonSet on special nodes.

We discuss it in Slack, and I found request of this feature in issue.

To start the discussion, I can suggest API-scheme for EnvoyProxyKubernetesProvider in EnvoyProxy (with Kubernetes Provider).

I propose the following architecture for implementation:

Now it's looks:

type EnvoyProxyKubernetesProvider struct {
	EnvoyDeployment *KubernetesDeploymentSpec `json:"envoyDeployment,omitempty"`

	EnvoyService *KubernetesServiceSpec `json:"envoyService,omitempty"`

	EnvoyHpa *KubernetesHorizontalPodAutoscalerSpec `json:"envoyHpa,omitempty"`
}

I think we can add field type and field for EnvoyDaemonset:

type KubernetesProviderType string

type EnvoyProxyKubernetesProvider struct {
        // By default - Deployment.
        Type KubernetesProviderType `json:"type,omitempty"`

	EnvoyDeployment *KubernetesDeploymentSpec `json:"envoyDeployment,omitempty"`

        EnvoyDaemonset *KubernetesDaemonsetSpec `json:"envoyDaemonset,omitempty"`

         // Need add logic to NOT create service
	EnvoyService *KubernetesServiceSpec `json:"envoyService,omitempty"`

        // Don't work, if type = Daemonset
	EnvoyHpa *KubernetesHorizontalPodAutoscalerSpec `json:"envoyHpa,omitempty"`
}

And:

// KubernetesDaemonsetSpec defines the desired state of the Kubernetes daemonset resource.
type KubernetesDaemonsetSpec struct {
	Patch *KubernetesPatchSpec `json:"patch,omitempty"`

	Pod *KubernetesPodSpec `json:"pod,omitempty"`

	Container *KubernetesContainerSpec `json:"container,omitempty"`

	InitContainers []corev1.Container `json:"initContainers,omitempty"`

         // Running only on special nodes
	NodeSelector map[string]string `json:"nodeSelector,omitempty"`
        Tolerations []corev1.Toleration `json:"tolerations,omitempty"`

        // By default - false
        HostNetwork bool `json:"hostNetwork,omitempty"`
}

(And need add logic for disable metrics Ports (19001) for DaemonSet ports). I think will be better, if we add field DisableMetricsPort to EnvoyProxyKubernetesProvider struct.

Finish

I can implement all the logic after agreeing on the API schema

[ShyunnY]

This feature looks great, and we probably need to discuss the design of the API.

[zvlb]

@ShyunnY When and in what format does the discussion of the API design? I would like to start implementing this functionality as soon as possible (it's very important for my work), however, I wouldn't want to waste time on something that won't be accepted.

[arkodg]

ptal @envoyproxy/gateway-maintainers @envoyproxy/gateway-reviewers

[zirain]

SGTM, but we need to talk about the API. In most cases, you can share the configuration between DS and Deployment, so maybe we can just something like mode?

here's what OpenTelemetry Operator did:

https://github.com/open-telemetry/opentelemetry-operator/blob/eaf998f2ae9ca93ecf319d6864690a2bd7cbe0c7/apis/v1beta1/opentelemetrycollector_types.go#L88

[arkodg]

Its looks like we've had multiple users requesting Daemonset + hostNetwork mode mode may get tricky for 2 cases

• we've already called it envoyDeployment
• we want to add a field to deployment but it may not exist in daemonset
• we may not want to spend extra cycles supporting all cases in daemonset that are supported in deployment since that is not the common case for this project

I'd suggest supporting a new envoyDameonSet knob with a single patch field, so we dont spend our efforts adding more fields into it

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[zvlb]

This feature merged in another PR - https://github.com/envoyproxy/gateway/pull/3092