Unable to deploy hnc to namespace other than hnc-system with `--enable-internal-cert-management`

[AustinSmart]

The webhook cert rotator uses a const referring to hnc-system as the source of the namespace that hnc is deployed into (ref 1, ref 2). I would expect references to resources consumed by hnc (hnc-webhook-server-cert in this case) should be made to the namespace in which the controller is deployed to by default, or at least be configurable.

This causes a crashloopbackoff of a deployment in a namespace other than hnc-system if the --enable-internal-cert-management flag is set.

{"level":"error","ts":1663007991.5142517,"logger":"cert-rotation","msg":"could not refresh cert on startup","error":"acquiring secret to update certificates: Secret \"hnc-webhook-server-cert\" not found","errorVerbose":"Secret \"hnc-webhook-server-cert\" not found\nacquiring secret to update certificates\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).refreshCertIfNeeded.func1\n\t/workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:215\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:220\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:233\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:226\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:421\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).refreshCertIfNeeded\n\t/workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:247\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).Start\n\t/workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:179\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/manager/runnable_group.go:218\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"}

[adrianludwin]

It doesn't surprise me that there are a few places where this is hardcoded. I'd be happy to look at PRs to make it configurable, but I'm not sure how to tell a pod what namespace it's running in (apart from through an env var or command line arg or something). Also, all the configs would have to be updated, I'm not sure if that's easy or hard offhand.

On Mon, Sep 12, 2022 at 2:53 PM Austin Smart @.***> wrote:

The webhook cert rotator uses a const referring to hnc-system as the source of the namespace that hnc is deployed into (ref 1 https://github.com/kubernetes-sigs/hierarchical-namespaces/blob/529259da83b27bf07191354789165767e1c3aaf3/internal/setup/webhooks.go#L26, ref 2 https://github.com/kubernetes-sigs/hierarchical-namespaces/blob/529259da83b27bf07191354789165767e1c3aaf3/internal/setup/webhooks.go#L38). I would expect references to resources consumed by hnc ( hnc-webhook-server-cert in this case) should be made to the namespace in which the controller is deployed to by default, or at least be configurable.

This causes a crashloopbackoff of a deployment in a namespace other than hnc-system if the --enable-internal-cert-management flag is set https://github.com/kubernetes-sigs/hierarchical-namespaces/blob/529259da83b27bf07191354789165767e1c3aaf3/cmd/manager/main.go#L103 .

{"level":"error","ts":1663007991.5142517,"logger":"cert-rotation","msg":"could not refresh cert on startup","error":"acquiring secret to update certificates: Secret "hnc-webhook-server-cert" not found","errorVerbose":"Secret "hnc-webhook-server-cert" not found\nacquiring secret to update certificates\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).refreshCertIfNeeded.func1\n\t/workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:215\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:220\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:233\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:226\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:421\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).refreshCertIfNeeded\n\t/workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:247\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).Start\n\t/workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:179\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/manager/runnable_group.go:218\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581 http://ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).refreshCertIfNeeded.func1%5Cn%5Ct/workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:215%5Cnk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1%5Cn%5Ct/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:220%5Cnk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext%5Cn%5Ct/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:233%5Cnk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection%5Cn%5Ct/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:226%5Cnk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff%5Cn%5Ct/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:421%5Cngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).refreshCertIfNeeded%5Cn%5Ct/workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:247%5Cngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).Start%5Cn%5Ct/workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:179%5Cnsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1%5Cn%5Ct/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/manager/runnable_group.go:218%5Cnruntime.goexit%5Cn%5Ct/usr/local/go/src/runtime/asm_amd64.s:1581"}

— Reply to this email directly, view it on GitHub https://github.com/kubernetes-sigs/hierarchical-namespaces/issues/227, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE43PZCUFSEFQPJYF6HFRGTV5532PANCNFSM6AAAAAAQKXWHJY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[AustinSmart]

Using the downward API fieldRef metadata.namespace to populate an env var would be a nice way.

It looks like the POD_NAMESPACE env var is being used in other places (ref 1, ref 2)

I will submit a PR if I have time.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.