Do you use kubemci command line tool? Tell us!

[nikhiljindal]

This is not an issue so much as a lightweight way of gathering information on who is using the kubemci command line tool. This is mostly to satisfy our curiosity, but might also help us decide how to evolve the project.

So, if you use kubemci for something, please chime in here and tell us more!

[nikhiljindal]

cc @glindste and @samanthakem Please feel free to post here once you start using it.

[glindstedt]

We've just recently started receiving some production traffic through an LB generated using kubemci. Currently we only have one cluster connected, but we will eventually have at least three (one per frontend region). For now our strategy for updating the LB is to generate a new LB bound to another IP that we run in parallel and just do a DNS switch. It's pretty straightforward though and should be a pretty rare operation. The main benefit for us is to be able to declaratively define an ingress resource and then get all the glue to connect the LB to multiple clusters in different regions under one IP for "free", so we don't have to manage it ourselves.

If you have any questions feel free to ask :)

[kinghrothgar]

I am in the process of moving my company's infrastructure to kube. We will have our production client website behind kubemci configured LBs as well as all of our internal tools.

[dicarlo2]

We are moving our production website over to a load balancer created with kubemci. HTTPS front-end, will probably move over to a https backend eventually. My biggest piece of feedback at this point would be to create thorough documentation - I sort of just winged it based on the tutorial, though it took me a while to figure out how to setup the certs (the pre-shared annotation is not documented anywhere). With that said, everything worked perfectly the first time - I'm pretty sure this is the only piece of kubernetes infra I've setup that actually worked exactly as I expected it to the first time.

[mdelio]

Hi @glindste, @kinghrothgar, and @dicarlo2; thanks for responding to this thread! I'm the product manager on multi-cluster and sit right next to Nikhil. We would love to setup some time, individually, to chat a little more about each your use-case to make sure we're addressing your longer-term needs. If you're open to it, could you send me and Nikhil an email at our Google.com addresses (mdelio at google.com and nikhiljindal at google.com)?

[wolmi]

We are now in the process of moving our production of WebRTC collaborative platform to three clusters under a LoadBalancer created with kubemci. Our goal is to put the platform closer to the end users and have a low latency communication.

Our main issue is related to managing websocket connections and certificates generated using cert-manager.

[ankurcha]

I saw that there is container native loadbalancing support using network endpoint groups. It would be nice to have that supported.

[pdecat]

Hi,

we are using kubemci to migrate publicly exposed workloads from zonal GKE public clusters to regional GKE private clusters with VPC native/Alias IP networking as those features cannot be enabled on existing clusters (e.g https://issuetracker.google.com/issues/70544277).

Those workloads are exposed by ingresses with external static IP addresses that must not change (referenced by some partners).

The conversion of existing GCE ingresses currently requires to:

• update Ingress class from gce to gce-multi-cluster in YAML manifest
• run kubemci create --force ... once to create ingress in new cluster and overwrite in old cluster (will fail because IP address is already in use by old Load Balancer)
• manually delete the old Global Load Balancer managed by ingress-gce to free the IP address
• then quickly re-run kubemci create --force ... several times until success.

Only a very few 502 errors were witnessed with this procedure in our tests.

It would be awesome if that conversion use case could be streamlined.

[michaelbannister]

I've just started looking at moving to container-native loadbalancing (using NEGs) but now concerned that it would block using multi-cluster ingress (which I'd hoped to do early next year). Also now looking at issue #196 which has been open for months with no indication of having been solved - this would definitely affect me. Issue #141 would also be a challenge as I rely heavily on wildcard certs (Lets Encrypt via cert-manager).

[nicholasklem]

We are using kubemci to loadbalance between 4 geographically spread kubernetes clusters since june. The ability to turn on global distributed loadbalancing so easy is amazing.

Some nitpicks: Updating ssl certs is a bit of a chore

• we have to keep ssl certs as updated kubernetes secrets on all clusters, no way of telling kubemci to pick from the first cluster.
• no way to use something like cert-manager to update certs and trigger reload

Solution is to buy certs for 1 year at the time, add them as kubernetes secrets under a new name, update manifest and run manually.

Also, the default timeouts of 10 times 60s is a gotcha. I would love to see support for BackendConfig like the standard ingress controller.

[lfallet]

Currently using kubemci in production, but we're facing a big issue with missing support of multiple TLS certificates. The instance group ingress annotation disappear sometimes, probably after some wrong call but can't understand why.

[necevil]

We are on in production. Liking it so far — if you aren't using a managed cert yet I would recommend swapping over, ended up being a lot less complicated than I thought and simplified the process in our case.

[rupeshkumar2010]

Have recently started exploring kubemci for multi cluster setup. but, I am getting below errors, while trying to specify multiple certificates in ingress.

Error ensuring SSL certs: googleapi: Error 400: Invalid value 'yih-google-certificate,yso-google-certificate,common-google-certificate'. Values must match the following regular expression: 'a-z?|[1-9][0-9]{0,19}', invalidParameter

• Error ensuring HTTPS target proxy: 1 error occurred:

TLS specific ingress configuration:

ingress.gcp.kubernetes.io/pre-shared-cert: "yih-google-certificate,yso-google-certificate,common-google-certificate"

note: trying to use google managed certificates

GKE version:

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-21T15:34:43Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.8-gke.10", GitCommit:"f53039cc1e5295eed20969a4f10fb6ad99461e37", GitTreeState:"clean", BuildDate:"2019-06-19T20:48:40Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}

[nikhiljindal]

Hello @rupeshkumar2010 Great to see that you have started exploring kubemci.

kubemci CLI does not support multiple TLS certs. More details and workarounds in https://github.com/GoogleCloudPlatform/k8s-multicluster-ingress/issues/120.

While I am here, thanks a lot everyone for providing great feedback and continuing to use the tool

[adam0292]

We are also attempting to use kubemci and also require multiple tls certificates. We would also like to take advantage of using Google's negs and backed service config to allow us to use session affinity and support CDN, session timeouts etc. I believe that these features are currently not supported. Is there any plan/roadman to add further enchancements to this tool?

[rupeshkumar2010]

Thanks @nikhiljindal for quick response.