argoproj
workflow step name in keyFormat for archiveLogs
Summary
I would like to use {{step.name}} in keyFormat definition when using archiveLogs: true.
Motivation
Currently the only way to separate the logs from a single workflow and preventing Argo from overwrite the logs from the preceding workflow steps I have to use {{pod.name}}. pod.name does not provide information about the step the log file was created by. This requires searching through all log files in each pod name folder to find the right output.
Proposal
When setting keyFormat: {{workflow.label.data-id}}/{{workflow.name}}-{{workflow.creationTimestamp}}/{{step.name}}.log Argo should create a single log file named after the steps name in that specified folder.
Example:
"A123/preprocessing-2020-07-08 12:11:53 +0000 UTC/merge-files.log"
This requires that:
- Argo detects "keyFormat" ends with ".log" so it does not create a subfolder with file "main.log"
- Argo is able to translate the variable "step.name".
Message from the maintainers:
If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.
Can I confirm, you simply want the log artifacts that are created and normall named main.log to be named after the step, and this should be done using keyFormat?
This is a little tricky to implement. The filename for the logs is set by the executor, but the executor does not know what step it is running. Instead the controller should set the log file name (or more specifically, the key).
It could do that by adding a environment variable named ARGO_LOG_KEY_FORMAT to the wait container:
https://github.com/argoproj/argo-workflows/blob/e28fb0744209529cf0f7562c71f7f645db21ba1a/workflow/controller/workflowpod.go#L514
Then, in executor.go rather than hardcoding to main.log, use that env var:
https://github.com/argoproj/argo-workflows/blob/e28fb0744209529cf0f7562c71f7f645db21ba1a/workflow/executor/executor.go#L560
Instead of overloading KeyFormat we should add a new LogKeyFormat to Config:
https://github.com/argoproj/argo-workflows/blob/master/pkg/apis/workflow/v1alpha1/artifact_repository_types.go
I think this should be added directly to the Config struct, not to S3/GCP/OSS etc.
There will need to be a new func in workflowpod.go to format the log key.
One consideration is that the log file name will not be fixed in the future. When running a container-set template, each container has a different name. So the default value should probably be {container.name}.log.
