kubeflow
[sdk] can't directly use image as a parameter in ContainerSpec (need to cast to string)
Environment
kfp 2.7.0 kfp-pipeline-spec 0.3.0 kfp-server-api 2.0.5
Steps to reproduce
from kfp import compiler, dsl
@dsl.container_component
def say_hello(image_uri: str):
"""Log a greeting and return it as an output."""
return dsl.ContainerSpec(
image=image_uri,
command=['echo'], args=['Hello']
)
@dsl.pipeline
def hello_pipeline(image_uri: str):
# greeting argument is provided automatically at runtime!
say_hello(image_uri=image_uri)
compiler.Compiler().compile(hello_pipeline, 'pipeline.yaml')
Error:
Traceback (most recent call last):
File "/home/glima/pipeline.py", line 15, in <module>
def hello_pipeline(image_uri: str):
File "/home/glima/.virtualenvs/kfp-env/lib/python3.9/site-packages/kfp/dsl/pipeline_context.py", line 65, in pipeline return component_factory.create_graph_component_from_func(
File "/home/glima/.virtualenvs/kfp-env/lib/python3.9/site-packages/kfp/dsl/component_factory.py", line 673, in create_graph_component_from_func
return graph_component.GraphComponent( File "/home/glima/.virtualenvs/kfp-env/lib/python3.9/site-packages/kfp/dsl/graph_component.py", line 68, in __init__
pipeline_spec, platform_spec = builder.create_pipeline_spec(
File "/home/glima/.virtualenvs/kfp-env/lib/python3.9/site-packages/kfp/compiler/pipeline_spec_builder.py", line 1919, in create_pipeline_spec build_spec_by_group(
File "/home/glima/.virtualenvs/kfp-env/lib/python3.9/site-packages/kfp/compiler/pipeline_spec_builder.py", line 1292, in build_spec_by_group
subgroup_container_spec = build_container_spec_for_task( File "/home/glima/.virtualenvs/kfp-env/lib/python3.9/site-packages/kfp/compiler/pipeline_spec_builder.py", line 608, in build_container_spec_for_task
pipeline_spec_pb2.PipelineDeploymentConfig.PipelineContainerSpec(
TypeError: bad argument type for built-in operation
Expected result
Pipeline compiles correctly
Materials and Reference
If I cast the image argument of ContainerSpec then it works
from kfp import compiler, dsl
@dsl.container_component
def say_hello(image_uri: str):
"""Log a greeting and return it as an output."""
return dsl.ContainerSpec(
image=str(image_uri),
command=['echo'], args=['Hello']
)
@dsl.pipeline
def hello_pipeline(image_uri: str):
# greeting argument is provided automatically at runtime!
say_hello(image_uri=image_uri)
compiler.Compiler().compile(hello_pipeline, 'pipeline.yaml')
Impacted by this bug? Give it a 👍.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.
I was trying to find the corresponding code (PipelineContainerSpec) but could not find it unfortunately
To me this seems to be a problem with how kfp compiles the pipeline. So I tried to workaround this with
def foo(image: str):
image2 = f"{image}"
return dsl.ContainerSpec(image=image2)
which made the pipeline compile. But the pipeline did not run. I think it is not possible to use input parameters for the image. The yaml looks like this:
exec-foo:
container:
command:
- sh
image: '{{$.inputs.parameters[''image'']}}'
but casting it as string like I did didn't work?
@dsl.container_component
def say_hello(image_uri: str):
"""Log a greeting and return it as an output."""
return dsl.ContainerSpec(
image=str(image_uri),
command=['echo'], args=['Hello']
)
Hi @glemarivero Thanks for your help! It compiles but the container wont start. If I hardcode the image, then it works. In the other issue someone wrote Only command and args support placeholders
mmm strange, we've been using this in production with no issues..are you using these versions?
kfp 2.7.0
kfp-pipeline-spec 0.3.0
kfp-server-api 2.0.5
How does the pipeline look like in the UI? And what values do you get when you go into the job?
In my case I get gcr.io/ml-pipeline/google-clou.. in the UI, but when I click on the job, under Container Location (Worker pool 0 (chief)) I get the one I passed as a parameter
Interesting. I am running
kfp 2.9.0
kfp-pipeline-spec 0.4.0
kfp-server-api 2.0.5
In the input parameter section I get the correct image that works when I use it hardcoded. But I dont get any logs. So I assumed that parametrized images for container components does not work
I am alos getting logs from kiverno:
Error node all-bwnmh.root.user-workload.executor: task 'all-bwnmh.root.user-workload.executor' errored: admission webhook "mutate.kyverno.svc-fail" denied the request: failed to add image information to the policy rule context: invalid image '{{$.inputs.parameters['image']}}' (bad image: docker.io/{{$.inputs.parameters['image']}}, defaultRegistry: docker.io, enableDefaultRegistryMutation: true: invalid reference format)
Which seems that the placeholder for the image did not get replaced
not sure if it's relevant but I'm using this in Vertex AI Pipelines, so maybe they've fixed it in their end
I see, maybe thats the reason, as I am running a self hosted kubeflow. Thank you nonetheless!
@HumairAK I am happy to contribute, would just need a pointer to the code. I guess this has to do with the templating of the pipeline spec?
@hahahannes apologies for the late reply, just saw this
If you're still interested in this, the offending code looks like it's here (when it's not casted). I think container specs are handled a bit differently than regular components in how parameters are resolved.
The issue is similar to https://github.com/kubeflow/pipelines/pull/11404, except here before it gets to convert_to_placeholder(task.container_spec.image) it has already been converted to an input value type, and the convert_to_placeholder doesn't know how to handle that. Maybe something as simple as a check for it's type and if it's input value just convert it to a string is sufficient.
Anyways that should give you a lead on how to handle the compilation side. As far as how this is handled in the backend, once a pipeline is submitted this still needs to be resolved. This will largely depend on how inputs are resolved for container types, if it's no different than components than the solution will likely utilize the helpers introduced as part of #11404 and make life easy for us here. You can confirm this by looking at the driver code changes in #11404 and debug this code once you have a pipeline yaml ready to submit with the string in the format {{$.inputs.parameters[''image'']}}.
Hopefully that gives you or the future assignee some leads, feel free to reach out here or slack (I'm a bit more responsive there).
The issue is similar to #11404, except here before it gets to
convert_to_placeholder(task.container_spec.image)it has already been converted to an input value type, and theconvert_to_placeholderdoesn't know how to handle that. Maybe something as simple as a check for it's type and if it's input value just convert it to a string is sufficient.
This is correct! I was able to fix by checking the type. Now the client side compilation works! @HumairAK
I was able to setup a development environment for the kubeflow pipelines api-server and I can upload pipeline and create runs. I hoped to reach the point where the container image get set but unfortunately, it is not clear yet how it is related to the driver (backend/srv/v2/driver). I see that the Argo Workflow gets prepared and I expect the image to be set there.
The issue is similar to #11404, except here before it gets to
convert_to_placeholder(task.container_spec.image)it has already been converted to an input value type, and theconvert_to_placeholderdoesn't know how to handle that. Maybe something as simple as a check for it's type and if it's input value just convert it to a string is sufficient.This is correct! I was able to fix by checking the type. Now the client side compilation works! @HumairAK
Would you mind telling me how to do that to test it out on my side? Thanks!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.