Support single controller deployments

[hbagdi]

Problem Statement

Currently the ingress controller is only capable of deploying configuration to one Kong Gateway at a time, the consequence of which is that to horizontally scale the data-plane in DBLESS mode you must create an ingress controller for each gateway which has some performance costs and wastes associated with it. The purpose of this issue is to make it possible to configure the KIC with multiple Gateways so that they all receive the same configuration updates.

Proposed Solution

• [ ] add functionality in the dataplane.KongClient to provide multiple Admin API endpoints and update each in c.Update()

Additional information

• Previous iterations of this issue described adding mTLS auth between the KIC and Gateways, but this was already resolved separately as part of #1958
• Horizontal scaling options where each Gateway has the same configuration is covered by this scope
• Multi-tenancy options where each Gateway has it's own separate configuration are not covered by this scope if you're interested in such a feature please file a separate issue

Acceptance Criteria

• [ ] as an end-user of the KIC when using DBLESS mode Kong Gateways as my data-plane I should be able to scale the data-plane horizontally with a single instance of the control-plane.
• [ ] for any number of data-planes, the KIC should be able to be configured with separate mTLS credentials for each
• [ ] the KEP has been updated and moves to "implemented" state: https://github.com/Kong/kubernetes-ingress-controller/blob/main/keps/0001-single-controller-multi-proxy.md

[seh]

We are definitely interested in this capability. If you need help, please let me know.

[marcoam]

+1

[shaneutt]

Plan:

1. proxy containers will now have their own pod in the Kong Ingress Controller
2. controller and proxies will have mTLS now that they will communicate over the container network and not over local host
3. service discovery
4. reconciliation needs to operate on N proxies now
5. include NetworkPolicy resources with KIC deployment which restrict access to the Kong Admin APIs now that they'll be exposed over the container network.

Notes:

• mTLS is on Kong side
• need to update kong/kong-operator, kong/charts

[shaneutt]

After discussing further with @mflendrich we're going to take some time to do some further investigation on the problem domain here (as opposed to going right into implementation as currently written) as we feel that there's some potential for large gains in rethinking our approach to this crossroads in how we deploy and manage the proxy.

[shaneutt]

The team has decided that we'll hold on this issue for now as we try to make some architectural changes to the KIC which will ultimately make implementing this easier.

[swapnilpotnis]

Plan:

1. proxy containers will now have their own pod in the Kong Ingress Controller
2. controller and proxies will have mTLS now that they will communicate over the container network and not over local host
3. service discovery
4. reconciliation needs to operate on N proxies now
5. include NetworkPolicy resources with KIC deployment which restrict access to the Kong Admin APIs now that they'll be exposed over the container network.

Notes:

• mTLS is on Kong side
• need to update kong/kong-operator, kong/charts

So does this mean, in future, kong would be able to support mTLS communication between kong <-> service?? If so, then could you please let us know if https://docs.konghq.com/kubernetes-ingress-controller/1.1.x/guides/upstream-mtls/ is something different than what is being implemented??

We basically have a use-case where we have an outside service communicating with kong running over AKS. Now we wish to establish mTLS between the two. As we have installed kong from helm charts, we were not able to accomodate the variables as mentioned in the above given link. Hence wanted to know if it was possible with the current kong release or is the mTLS support by kong yet to be implemented?

[shaneutt]

So does this mean, in future, kong would be able to support mTLS communication between kong <-> service??

In this particular case:

This covers mTLS between the Kubernete Ingress Controller Pod and access to the Admin API in the Kong Proxy Pod. Note that historically the containers have been in a single pod together and used localhost connectivity so mTLS was not enabled.

This does not pertain to any other services (e.g. the services being proxied to) for this scope.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[shaneutt]

Several weeks ago we started some work on this but it was not finished due to a desire by the maintainers to wait until KIC 2.0 which will bring a major re-architecture that would better support this feature.

We'll consider this blocked on the KIC 2.0 milestone for now and revisit once that is completed.

[shaneutt]

As stated before we're still considering the potential synergy with a future KIC operator, but we're also now considering the synergy of this improvement with the advent of Gateway support. The relevant KEP is being re-opened and modernized to reflect recent developments: https://github.com/Kong/kubernetes-ingress-controller/pull/2256

[shaneutt]

Relevant to the mTLS portion of this, flags have been added as part of #1958 to provide client certs for mTLS with the Kong Gateway. The remaining portion of the work which relates to being able to support multiple Gateways with a single KIC is still covered by this issue and the KEP https://github.com/Kong/kubernetes-ingress-controller/blob/main/keps/0001-single-controller-multi-proxy.md.

[mflendrich]

One option to consider in this spike as raised by @rainest: use alternative control plane solutions to achieve this problem.

[ludovic-pourrat]

Hi,

Within this KEP https://github.com/Kong/kubernetes-ingress-controller/blob/main/keps/0001-single-controller-multi-proxy.md, will the support of the hybrid mode with the dbless feature is an option ?

[shaneutt]

Hi,

Within this KEP https://github.com/Kong/kubernetes-ingress-controller/blob/main/keps/0001-single-controller-multi-proxy.md, will the support of the hybrid mode with the dbless feature is an option ?

With hybrid mode you don't generally need more than one ingress controller (if you're using the ingress controller at all), so currently hybrid mode isn't really a prominent part of the direction so far. This feature would in fact represent an alternative to hybrid mode, at least from the perspective of providing horizontal scaling options for the dataplane (kong gateway).

Was there something specific you needed or were hoping for?

[ludovic-pourrat]

@shaneutt thanks for you reply, and what we are hoping for is to run the hybrid mode with the dbless feature (with one ingress controller instance, one or two control planes and a brunch of data planes), which is not supported at the moment. Secondly from an availability point of view, the hybrid mode seem's more capable to handle restart and failures of the data plane, where in the current dbless implementation it takes more time for the data plane to resurface with a valid setup. In a sense this was about taking benefit from the current hardening of the hybrid mode but with the dbless feature and with better resiliency of the control plane if this kep where taking the hybrid mode as an option.

[shaneutt]

@shaneutt thanks for you reply, and what we are hoping for is to run the hybrid mode with the dbless feature (with one ingress controller instance, one or two control planes and a brunch of data planes), which is not supported at the moment. Secondly from an availability point of view, the hybrid mode seem's more capable to handle restart and failures of the data plane, where in the current dbless implementation it takes more time for the data plane to resurface with a valid setup. In a sense this was about taking benefit from the current hardening of the hybrid mode but with the dbless feature and with better resiliency of the control plane if this kep where taking the hybrid mode as an option.

I see. This issue isn't related to hybrid mode so it would seem we should take this conversation elsewhere. Would you please put in a feature request for the specific requirements you have as a separate issue?

[ludovic-pourrat]

@shaneutt sure thx !

[mflendrich]

May be implemented by https://github.com/Kong/kubernetes-ingress-controller/issues/3209

[pmalek]

This is currently being worked on.

#3268 aims to enable rendering and sending configs for multiple Kong Gateways: this will be done via the same flag that's in place today --kong-admin-url (by passing multiple, comma separated values).

When that's done then we'll work on making this not be bound to controller's Pod's lifetime, specifically to use a label selector for Kong's Admin API Service.

If all that gets released we'll be able to then incorporate those changes into helm chart so that users can utilize that deployment method to deploy single controller with multiple Gateways.

[mflendrich]

The remaining work here is to complete all the remaining checkboxes in the AC.

[czeslavo]

We still have

For any number of data-planes, the KIC should be able to be configured with separate mTLS credentials for each

in ACs which didn't have its tracking issue. I created one, but I'm not sure how would that work having gateways discovered dynamically via a headless service. 🤔 Did we try to design this anywhere?

[pmalek]

We still have

For any number of data-planes, the KIC should be able to be configured with separate mTLS credentials for each

in ACs which didn't have its tracking issue. I created one, but I'm not sure how would that work having gateways discovered dynamically via a headless service. 🤔 Did we try to design this anywhere?

For posterity: as agreed on sync meeting #3603 will track this effort but it won't be land in v2.9.

[mflendrich]

The core has been implemented and will ship in KIC 2.9 and the Helm chart 2.17.

There are a few open items left:

• #3401
• #3618
• #3423
• #3619
• #3568
• #3592
• #3606
• #3604

Now that we have only those details to polish, I'm closing this huge issue in favor of the more targeted ones above.