pipeline
pipeline copied to clipboard
tektoncd
Feature Request: Using ConfigMap Values For Params
Expected Behavior
When defining parameters for a Task in my Pipeline, I would like to use values for the parameters which I defined in a ConfigMap, instead of hardcoding them in the Pipeline.
A cool solution would look like the use of ConfigMaps when defining environment variables in a Pod, there I am able to do this:
env:
- name: SOME_ENV_VARIABLE
valueFrom:
configMapKeyRef:
name: my-config
key: my-key
if params could work the same way it would be like this:
params:
- name: some-parameter
valueFrom:
configMapKeyRef:
name: pipeline-config
key: some-key
Actual Behavior
If I want to use a ConfigMap I can only use it on a Step level, and I can't directly access its values, this becomes an issue in the following Task:
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
name: pmd-test
spec:
inputs:
resources:
- name: git-source-code
type: git
params:
- name: image
type: string
description: The image that will be used in the test
default: rawdee/pmd
- name: command
type: string
description: The command that will be used in the test
default: pmd
- name: args
type: string
description: The args that will be used in the test
default: -language java -dir /workspace/git-source-code -rulesets ./my-rulesets.xml
steps:
- name: test-pmd
image: $(input.params.image)
command: ["$(input.params.command)"]
args: ['$(input.params.args)']
in this example, I want to have the possibility configure the entire Step based on the ConfigMap, but currently it would look like this:
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
name: pmd-test
spec:
volumes:
- name: config-volume
configMap:
name: pipeline-config
inputs:
resources:
- name: git-source-code
type: git
steps:
- name: test-pmd
volumeMounts:
- name: config-volume
mountPath: /my-path
image: $(cat /my-path/my-image)
command: ["$(cat /my-path/my-command)"]
args: ['$(cat /my-path/my-args)]
Note that this wouldn't work for the image since I'm only able to use variable substitution there.
+1, we should make this change. This makes configuring the controller more standard, and could let users update the support images to use (git-init, creds-init, etc.) without requiring a restart of the controller pod.
Thinking through this a bit more:
Luckily at least some of the groundwork is already done: support images are parsed from flags and set as fields on the pipeline.Images type.
These fields should become methods that return the current value, and some background process could monitor the ConfigMap and update the values returned by that method.
As a first step we can update one image to use the ConfigMap style, then prove it works, then update the rest. It probably wouldn't be a very hard change at all.
Knative configures their queue proxy image using a ConfigMap: https://github.com/knative/serving/blob/b3079512daae9708f11c7b8ea41ef54023c91725/config/config-deployment.yaml#L26
Their pkg/deployment/config.go returns a *Config with a QueueSidecarImage field that gets updated when the ConfigMap changes.
Their "deployment config" object is effectively the same as our "pipeline.Images" object.
sigh -- and after I just closed #1197 . The case can be made for this feature however I can say that I really thought I wanted this feature but in practice found I never needed it.
I'm not sure I understand how these are related?
That issue seems to be about letting users declare TaskRun/PipelineRun values as coming from a ConfigMap (which I admit does sound useful?).
This issue is about letting operators specify/update controller args using ConfigMaps, which is more of an operator concern and not really exposed to end users.
Am I misunderstanding?
Hmm... maybe I'm the one misunderstanding. @tomfrenken Is this issue for something like an install time kustomization of a Task and Pipeline. e.g. not something at runtime? Can you maybe describe a bit more what you're looking for.
@skaegi I think @ImJasonH sums it up very well, that's precisely what I was meaning But to answer your question a little better, yes in my case I thought about customization at install time.
I’m really interested in that feature. I don’t know if it’s out of scope here but I think it could be great to be able to get the default value from the ConfigMap. It would mean having a way to deploy Tasks and Pipelines which could be invoked by operators via ConfigMaps and also by user via the CLI.
Hey folks, I'm coming late to this issue but I was wondering if someone could clearly articulate use-cases and scope of this change. The description sounds to me exactly like @skaegi 's valueFrom issue and customization at install time isn't filling in the gaps for me. For example, from the issue description the following is stated as being desirable:
params:
- name: some-parameter
valueFrom:
configMapKeyRef:
name: pipeline-config
key: some-key
But then we seem to says this is not what's being asked for:
That issue seems to be about letting users declare TaskRun/PipelineRun values as coming from a ConfigMap (which I admit does sound useful?). This issue is about letting operators specify/update controller args using ConfigMaps, which is more of an operator concern and not really exposed to end users.
And now I'm really confused. I see no mention of the controller or its arguments anywhere in the initial issue description. This issue really reads to me as precisely valueFrom as described in @skaegi's linked issue. So I'm clearly missing something, either from a gap in my own understanding or a gap in the way this issue is articulated.
So before we move ahead could somebody write out one or two clearly defined user-facing use cases and a description of what's in-scope / out-of-scope? It would be excellent to see what the intended final syntax looks like as well in the context of a Task / TaskRun / Pipeline / PipelineRun. I don't mind if it's a short design doc or just a comment here with some code blocks.
Also: the title of this issue is literally "Feature Request: Using ConfigMap Values For Params". I am so confused.
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
/lifecycle rotten
Send feedback to tektoncd/plumbing.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
/lifecycle stale
Send feedback to tektoncd/plumbing.
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
/close
Send feedback to tektoncd/plumbing.
@tekton-robot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity. Reopen the issue with
/reopen. Mark the issue as fresh with/remove-lifecycle rotten./close
Send feedback to tektoncd/plumbing.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
I just discovered this issue, for me this comes into play when I do not control the task, i.e. it's a ClusterTask or a Task being sourced from elsewhere. This task is essentially a black box that specifies parameters x,y,z and I would like to source them from a configmap or a secret, passing environment variables here isn't helpfull since the task is specifically expecting parameters.
The only way I can see to achieve this is to write a pre-task that sources the configmap or secret and outputs the values as results which can then be referenced as parameters. This feels overly convoluted and awkward, the original syntax that is being proposed here fits the bill of what I i'm looking for, i.e.:
params:
- name: some-parameter
valueFrom:
configMapKeyRef:
name: pipeline-config
key: some-key
/reopen I think this is still something you may want to pursue in some way or another.
@vdemeester: Reopened this issue.
In response to this:
/reopen I think this is still something you may want to pursue in some way or another.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
I can provide a use case we are facing that this feature would solve.
We have a pipeline Task that builds a Java application using Gradle. In order to have a speedy execution, we have built an image containing all Gradle dependencies that we use in this Task. The Task looks like this now:
apiVersion: tekton.dev/v1beta1
kind: Task
...
spec:
steps:
image: "our.image.registry/gradle-cached-image:1.4.5"
...
Since the dependencies of our Java application regularly change, we automatically re-build this image and update the Task definition by hand.
It would be super useful if we could just update a ConfigMap in k8s, containing the new image name, and in the Task put something like:
kind: Task
spec:
params:
- name: gradle-image
valueFrom:
configMapKeyRef:
name: gradle-image
key: image
steps:
image: "$(params.gradle-image)"
@chengjoey @vdemeester @lbernick Hi, I ran into this issue and it seems the first attempt to implement the feature stopped due to the lack of TEP. May I take it over from there and start writing a TEP? 🙂
Definitely, thanks @shuheiktgw! You can find more info on the TEP process and how to create one here
Actually, it looks like there's an existing TEP for this: https://github.com/tektoncd/community/pull/868/ It may be worth checking in with the author to see whether they are still working on it.
I would really like this feature, use-case as follows.
We're trying to inject a SONARQUBE_TOKEN into the default (installed by the openshift-pipelines operator) s2i-java ClusterTask as a parameter. We do not want to hard-code the token into the pipeline params as we want to keep the repo containing the pipeline free from secrets. And we also do not want to fork the ClusterTask just to be able to add a valuefrom.secretkeyref-env binding.
apiVersion: tekton.dev/v1beta1
kind: Pipeline
spec:
tasks:
- name: s2i-build
params:
- name: ENV_VARS
value:
- >-
SONARQUBE_TOKEN=wouldBeCoolToFetchFromCMOrSecret
taskRef:
kind: ClusterTask
name: s2i-java
