Dynamically mapping Task inputs to Artifact fields

[Trollgeir]

I'm trying to rewrite some yml-files to hera, and I've stumbled upon some issues related to mapping an input variable (a google cloud storage file path) to the parameters of an InputArtifact (GCSArtifact).

In the yml-workflow this is done by templating:

inputs:
    parameters:
      - name: bucket
      - name: filename
    artifacts:
      - name: file
        path: /file_to_process
        gcs:
          bucket: '{{inputs.parameters.bucket}}'
          key: '{{inputs.parameters.filename}}'

What's the intended way to use Hera to accomplish this? Should I include templating in the artifact fields, like this?

GCSArtifact(name="file", 
            path="/file_to_process",
            bucket='{{inputs.parameters.bucket}}', 
            key='{{inputs.parameters.filename}}')

I'm asking primarily because I've yet to see any templating in the examples.

[Trollgeir]

Given the example shown here, I reckon this is actually the right way to implement this. I was under the naive assumption that templates wouldn't be needed with Hera, but I realize that there's really no other way to pass variables through the workflow dynamically without it.

[flaviuvadan]

@Trollgeir what do you think about a ParameterizedArtifact that has fields set via inputs.parameters and users specify the key of that dictionary? I think Hera should be as wary as possible of requiring users to type out the '{{inputs.parameters.filename}}' syntax explicitly. It makes working with Argo a bit challenging. Also, there are many users who like Hera specifically because it does not require them to use that syntax and will set it for them.

[Trollgeir]

@Trollgeir what do you think about a ParameterizedArtifact that has fields set via inputs.parameters and users specify the key of that dictionary? I think Hera should be as wary as possible of requiring users to type out the '{{inputs.parameters.filename}}' syntax explicitly. It makes working with Argo a bit challenging. Also, there are many users who like Hera specifically because it does not require them to use that syntax and will set it for them.

I like the mindset of this idea. After contemplating this for a bit, I was thinking if this could be done even more generic. For example, what if all the fields of the Hera models (which should be an instance of BaseModel) could always support an extra type named something like Parameter (like InputParameter or GlobalInputParameter) that we can resolve either at creation or in a context. This way, we could set bucket = InputParameter("bucket").

I'm not 100% on-board on my own suggestion myself, as I think there are some collisions with pre-existing concepts which will probably lead to breaking changes.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.