Got errors on kubebuilder v3 quick start

[astraw99]

What broke? What's expected?

Using the latest kubebuilder cli, quick start a new project based on the official document, got these errors:

1. Enable the leader-election, then make run, got error:

go fmt ./...
go vet ./...
go run ./main.go
1.665666448465308e+09   ERROR   setup   unable to start manager {"error": "unable to find leader election namespace: not running in-cluster, please specify LeaderElectionNamespace"}
main.main
...
exit status 1
make: *** [run] Error 1

2. After make deploy IMG=xxx/xxx:tag, the controller-pod got error:

1.6656687258729894e+09  ERROR   controller-runtime.client.config        unable to get kubeconfig        {"error": "open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied"}
sigs.k8s.io/controller-runtime/pkg/client/config.GetConfigOrDie
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/client/config/config.go:153
main.main
        /workspace/main.go:68
runtime.main
        /usr/local/go/src/runtime/proc.go:250

3. As the doc said Important: Run "make" to regenerate code after modifying this file, but make did not generate latest CRD fields.

Reproducing this issue

Just quick start a new project based on the official document, then will reproduce this issue.

KubeBuilder (CLI) Version

Version: main.version{KubeBuilderVersion:"3.7.0", KubernetesVendor:"1.24.1", GitCommit:"3bfc84ec8767fa760d1771ce7a0cb05a9a8f6286", BuildDate:"2022-09-20T17:21:57Z", GoOs:"darwin", GoArch:"amd64"}

PROJECT version

No response

Plugin versions

No response

Other versions

No response

Extra Labels

No response

[laxmikantbpandhare]

make run is working for me. But I haven't updated any code for the leader election.

laxmikantbhaskarpandhare@lpandhar-mac kb % make run
test -s /Users/laxmikantbhaskarpandhare/comparison-memcached-operator/remove/kb/bin/controller-gen || GOBIN=/Users/laxmikantbhaskarpandhare/comparison-memcached-operator/remove/kb/bin go install sigs.k8s.io/controller-tools/cmd/[email protected]
/Users/laxmikantbhaskarpandhare/comparison-memcached-operator/remove/kb/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/Users/laxmikantbhaskarpandhare/comparison-memcached-operator/remove/kb/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./main.go
1.665674102135634e+09	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
1.665674102135971e+09	INFO	setup	starting manager
1.665674102136213e+09	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
1.665674102136213e+09	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.6656741021363149e+09	INFO	Starting EventSource	{"controller": "guestbook", "controllerGroup": "webapp.my.domain", "controllerKind": "Guestbook", "source": "kind source: *v1.Guestbook"}
1.665674102136364e+09	INFO	Starting Controller	{"controller": "guestbook", "controllerGroup": "webapp.my.domain", "controllerKind": "Guestbook"}
1.665674102236641e+09	INFO	Starting workers	{"controller": "guestbook", "controllerGroup": "webapp.my.domain", "controllerKind": "Guestbook", "worker count": 1}

[laxmikantbpandhare]

May I know what changes you did to the leader election? For this - Enable the leader-election

[astraw99]

May I know what changes you did to the leader election? For this - Enable the leader-election

On generated main.go L57, just set leader-election true.

flag.BoolVar(&enableLeaderElection, "leader-elect", true, ...

[everettraven]

1.665666448465308e+09 ERROR setup unable to start manager {"error": "unable to find leader election namespace: not running in-cluster, please specify LeaderElectionNamespace"}

Using make run runs the controller as a process external to the cluster (on your local machine). Encountering this error when setting leader election and then running outside of the cluster makes sense to me.

1.6656687258729894e+09 ERROR controller-runtime.client.config unable to get kubeconfig {"error": "open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied"}

This error is a bit different. I'm not sure why it would be getting a permission denied when attempting to load the kubeconfig.

It seems possible it could be related to ServiceAccounts no longer auto-generating tokens by default as of kubernetes 1.24 (see changelog here: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#urgent-upgrade-notes).

If you update the config/rbac/service_account.yaml file and add this to the end of it:

automountServiceAccountToken: true

does running make deploy IMG=... end up working?

[everettraven]

After running through the tutorial locally and using make deploy IMG=... I was unable to replicate the error:

1.6656687258729894e+09 ERROR controller-runtime.client.config unable to get kubeconfig {"error": "open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied"}

I even attempted it after forcing the --leader-elect flag to default to true as mentioned:

May I know what changes you did to the leader election? For this - Enable the leader-election

On generated main.go L57, just set leader-election true.

flag.BoolVar(&enableLeaderElection, "leader-elect", true, ...

I did not need to do:

If you update the config/rbac/service_account.yaml file and add this to the end of it:

automountServiceAccountToken: true

does running make deploy IMG=... end up working?

[liggitt]

It seems possible it could be related to ServiceAccounts no longer auto-generating tokens by default as of kubernetes 1.24 (see changelog here: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#urgent-upgrade-notes).

If you update the config/rbac/service_account.yaml file and add this to the end of it:

automountServiceAccountToken: true

That should not be necessary. Service accounts still mount tokens by default in 1.24, they just do so using ephemerally generated tokens, not tokens stored in Secret API objects

[astraw99]

After running through the tutorial locally and using make deploy IMG=... I was unable to replicate the error:

My cluster K8s version is:

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:25:06Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}

Kubebuilder version is:

Version: main.version{KubeBuilderVersion:"3.7.0", KubernetesVendor:"1.24.1", GitCommit:"3bfc84ec8767fa760d1771ce7a0cb05a9a8f6286", BuildDate:"2022-09-20T17:21:57Z", GoOs:"darwin", GoArch:"amd64"}

With these context, will replicate the error. @everettraven

[camilamacedo86]

Hi @liggitt,

The following steps are not accurate.

go fmt ./...
go vet ./...
go run ./main.go
1.665666448465308e+09   ERROR   setup   unable to start manager {"error": "unable to find leader election namespace: not running in-cluster, please specify LeaderElectionNamespace"}
main.main 

Could you please perform the suggested steps to check if the error is still faced?

[astraw99]

May I know what changes you did to the leader election? For this - Enable the leader-election

On generated main.go L57, just set leader-election true.

flag.BoolVar(&enableLeaderElection, "leader-elect", true, ...

@camilamacedo86 I modified this first, and then make install, make run, then will encounter the LeaderElectionNamespace error. PTAL thanks.

[camilamacedo86]

Hi @astraw99,

I see. If you enable the leader election you are facing the problem {"error": "unable to find leader election namespace: not running in-cluster, please specify LeaderElectionNamespace"}. Then:

a) Is the issue faced when/if you update the config/manager.yaml to pass the flag == true and deploy the project on the cluster as well? OR the namespace is only required if/when you run it outside of the cluster for developing/test purposes with make run?

PS.: Note that in the quick start we do not have any step saying for someone to enable the leader election. Therefore, the title/description here does not seem accurate. It is more like so far "default scaffold does not work if/when we enable leader election and we use make run to test the project outside of the cluster"

[astraw99]

Hi @astraw99,

I see. If you enable the leader election you are facing the problem {"error": "unable to find leader election namespace: not running in-cluster, please specify LeaderElectionNamespace"}. Then:

a) Is the issue faced when/if you update the config/manager.yaml to pass the flag == true and deploy the project on the cluster as well? OR the namespace is only required if/when you run it outside of the cluster for developing/test purposes with make run?

PS.: Note that in the quick start we do not have any step saying for someone to enable the leader election. Therefore, the title/description here does not seem accurate. It is more like so far "default scaffold does not work if/when we enable leader election and we use make run to test the project outside of the cluster"

Yes, it is the make run to test the project outside of the cluster got the leader-election namespace error. Raised a PR https://github.com/kubernetes-sigs/kubebuilder/pull/3021 to fix it.

[camilamacedo86]

HI @astraw99,

Is the same error faced using make deploy IMG or not?

[astraw99]

HI @astraw99,

Is the same error faced using make deploy IMG or not?

No, make deploy into the cluster works well. Only faced the sa permission error, opened a new issue #3028 to clarify it.

[jmrodri]

/label triage/accepted

[camilamacedo86]

Hi @astraw99,

Only faced the sa permission error, opened a new issue https://github.com/kubernetes-sigs/kubebuilder/issues/3028 to clarify it.

The default scaffold does have not SA. So, how can that can be faced by default?

However, this issue was used to just add the manifest target to the make build target (https://github.com/kubernetes-sigs/kubebuilder/pull/3020). Let's discuss the other points in the https://github.com/kubernetes-sigs/kubebuilder/issues/3028