kubernetes-sigs
config/* files can't be kustomized as-is w/o envsubst
In recent times it seems that feature-gate flags were added to CAPI controllers. The values for these flags seem to be assigned by envsubst variable expansion e.g. ${EXP_SOMETHING:=false}. This convention renders the config/* manifests not usable as-is using only kustomize, and requires an additional tool, envsubst, in order to properly generate kustomized manifests.
I'd like for CAPI to eliminate this additional binary requirement of envsubst. It seems that in the CAPI codebase that the feature gate flags are used by dev and CI. I'm hoping there's another way to accomplish the same thing in order to eliminate the dependency on envsubst.
A somewhat related issue is #3252, but that has more to do with cluster templates.
/cc @fabriziopandini
@jdef thanks for this report IMO fixing this is in the scope of the work of the management cluster operator, given that it is redesigning how the user can configure providers in a fully declarative way. I have added a comment on the proposal to make https://github.com/kubernetes-sigs/cluster-api/pull/3833/files#r512598988 to make sure this is in the radar cc @wfernandes
WRT to the current problem,
This convention renders the config/* manifests not usable as-is using only kustomize, and requires an additional tool...
I know that kustomize can still be used over config/* because we are still doing in several places (config/e2e); I'm wondering if, as a temporary workaround, you can add a patch that replaces the lines where ${EXP_SOMETHING:=false} is used with the actual values. Not ideal, but it should work...
I know that kustomize can still be used over config/*
To be clear, my expectation is that one should be able to successfully execute a pipeline along the lines of:
:; kustomize build config | kubectl apply -f -
.. and this is currently not possible given the additional envsubst (or similar) requirement.
gotcha, this is where the additional patch could help (to remove the need of envsubst)
I think the problem with this ask is that it would mean that this feature flags (EXP_MACHINE_POOL, etc) would be set at manifest creation time, whereas with the flow we document in the quick-start tutorial we support turning these features on at deploy time.
One way to get around that might be to have multiple kustomize overlays, one of which is the deferred/envsubst version, but others of which resolve to fully finalized yaml. I think we'll still get into the problem with kustomize of needing one overlay per feature, or an overlay that is "all experiments turned on" or something.
kustomize recently added "components" (kind of like patchset mixins). maybe that could help? https://github.com/kubernetes-sigs/kustomize/blob/master/examples/components.md
Yeah I was looking at this, I think we could do a component like machinepools that inserts the --feature-gates=MachinePool=true argument onto the deployment, but then you still have to incorporate that component into an overlay, so you end up in the position of needing a ton of overlays, at least if you want to give your users that same flexibility of turning on each component individually.
This would also mean changing the way CAPI releases its YAML to be something like
- cluster-api-components.yaml
- cluster-api-components-with-machinepools.yaml
- cluster-api-components-with-clusterresourcesets.yaml
- cluster-api-components-with-clusterresourcesets-and-machinepools.yaml
Summary of a recent discussion about this:
- the management cluster operator work is going to remove the need of variables for feature flags
- the work on multitenancy could potentially remove the need of variables for credentials (to be confirmed)
- at least in Azure, there is still the need for env vars for AZURE_ENVIRONMENT
at least in Azure, there is still the need for env vars for AZURE_ENVIRONMENT
@nader-ziada @devigned wdyt about moving AZURE_ENVIRONMENT to AzureCluster spec so it's no longer needed in the CAPI yaml? If subID/credentials are per cluster, there's no reason Azure Cloud shouldn't also be per cluster.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle stale - Mark this issue or PR as rotten with
/lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle stale - Mark this issue or PR as rotten with
/lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle stale - Mark this issue or PR as rotten with
/lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
/triage accepted /help-wanted
we have to re-assess the situation and see if we can remove all the variables from those files as suggested above
before acting we should reconcile requirements with https://github.com/kubernetes-sigs/cluster-api/issues/3247 that advocates for more templating features into provider yaml files
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle stale - Mark this issue or PR as rotten with
/lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
/lifecycle frozen /help
even if there are conflicting requirements between this issue and https://github.com/kubernetes-sigs/cluster-api/issues/3247, finding a way forward could help people using Cluster API without cluterctl
@fabriziopandini: This request has been marked as needing help from a contributor.
Guidelines
Please ensure that the issue body includes answers to the following questions:
- Why are we solving this issue?
- To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
- Does this issue have zero to low barrier of entry?
- How can the assignee reach out to you for help?
For more details on the requirements of such an issue, please see here and ensure that they are met.
If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.
In response to this:
/lifecycle frozen /help
even if there are conflicting requirements between this issue and https://github.com/kubernetes-sigs/cluster-api/issues/3247, finding a way forward could help people using Cluster API without cluterctl
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
This issue is labeled with priority/important-soon but has not been updated in over 90 days, and should be re-triaged.
Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release.
You can:
- Confirm that this issue is still relevant with
/triage accepted(org members only) - Deprioritize it with
/priority important-longtermor/priority backlog - Close this issue with
/close
For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-triage accepted
This issue has not been updated in over 1 year, and should be re-triaged.
You can:
- Confirm that this issue is still relevant with
/triage accepted(org members only) - Close this issue with
/close
For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-triage accepted
/close
Unfortunately this issue is not getting traction since one year now, and I don't see the project moving away from clusterctl and envsubst as a primary method to deploy providers.
if the issue arises again we can eventually consider adding a make target to generate GitOps friendly manifests Also, the https://github.com/kubernetes-sigs/cluster-api-operator is quickly approaching 1.0 and it is specifically designed to provide a GitOps friendly approach without clusterctl (and envsubst)