SDK forces the usage of namespace when running in kubernetes

[tdeheurles]

Overview of the Issue

When I run golang in kubernetes and use the golang SDK to register a service, it forces the usage of "ns" in the query (which is a Consul Enterprise feature). This results in the call failing : Unexpected response code: 400 (Bad request: Invalid query parameter: "ns" - Namespaces are a Consul Enterprise feature).

The Namespace in Config is set automatically when I'm not setting it, but I would like it to not be set so that ns=... doesn't appear in the query url.

When I run my code locally:

• to a consul server started locally: there is no issue, the service is registered
• to a consul running in kubernetes (started with hashicorp helm chart) and executing my command through kubectl port-forward from my machine: there is no issue, the service is registered

When I run my code from inside kubernetes:

• to a consul running in kubernetes with my registration executed from another kubernetes namespace: the issue appears, golang SDK automatically set namespace to the consul's kubernetes namespace.

Reproduction Steps

1. Start consul from the hashicorp helm chart (see below for details)
2. Execute a port-forward to a local machine and confirm consul golang sdk works as expected by registering a service
3. dockerize the validated code from previous step and run it from inside kubernetes. You should now see the code failing with the namespace issue

Start consul in k8s

        helm upgrade --install \
            --namespace="consul-specific" \
            --set="global.name=consul" \
            --set="server.replicas=3" \
            --version="0.32.1" \
            "consul" . 

Operating system and Environment details

• The code run successfully from a windows 11 WSL.
• The code fails from inside kubernetes on linux standard EKS nodes

Log Fragments

Unexpected response code: 400 (Bad request: Invalid query parameter: "ns" - Namespaces are a Consul Enterprise feature)

[Amier3]

Hey @tdeheurles

Thanks fo bringing this to our attention, i'll need to transfer this to our consul-k8s repo since it seems to be kubernetes specific

[tdeheurles]

Hey @Amier3 , thanks for help. Would you have a link where I should follow in consul-k8s ? Or de we continue discussing here ?

[lkysow]

Hi @tdeheurles can you show us your Golang code?

[tdeheurles]

Hey @lkysow, this code, when executed from k8s is failing for me:

package main

import (
	"github.com/hashicorp/consul/api"
)

func main() {

	client, err := api.NewClient(&api.Config{
		Address: "consul-server.cartel-consul:8500",
	})
	if err != nil {
		panic(err)
	}

	reg := &api.AgentServiceRegistration{
		Name: "cartel-env-kind",
		ID:   "cartel-env-kind-0",
	}

	if err := client.Agent().ServiceRegister(reg); err != nil {
		panic(err)
	}
}

results in:

panic: Unexpected response code: 400 (Bad request: Invalid query parameter: "ns" - Namespaces are a Consul Enterprise feature)

goroutine 1 [running]:
main.main()
        /project/components/svc_consul/397b2034__execution__k8s_issue_demo.go:22 +0x10e
exit status 2

[lkysow]

Are you running this from a regular pod in Kubernetes or one part of the service mesh?

[lkysow]

I just tried to reproduce this by building a Go binary, kubectl cp'ing to a random pod and execing it and I didn't get the same error. What is different with how you're seeing this error?

[tdeheurles]

Where are you running consul ? in the same namespace as your client pod ?

[tdeheurles]

I'm running without mesh, "standard" EKS k8s I could say.

[lkysow]

No, in separate namespaces. Can you paste your deployment/pod definition too?

[tdeheurles]

Hey @lkysow , (sorry for absence, some holidays :-) )

What are you looking for ? At the moment I was doing debug by starting a manual pod (kubectl run) and just kubectl cp the code into it. Do you see any elements that could interfere with this ?

I'm happy to do a session with you on this if you wish.

[tdeheurles]

Any news on this ? Which versions (k8s/consul) did you use to have it working fine @lkysow please ?

[t-eckert]

This is an interesting problem. I gave it a shot today and couldn't replicate the issue. Can you try it with Consul on Kubernetes 0.49.0 and let me know if you still see the issue? I'd also be interested in what you are passing to the run command as your testing pod container. I just used the Golang Docker image and it worked fine.

[david-yu]

Closing as no response for a month. Please try the latest versions of Consul and Consul K8s to give us a better idea of how this issue is occurring.