argo-workflows icon indicating copy to clipboard operation
argo-workflows copied to clipboard
verified publisher icon argoproj

HTTP template treated as container template and fails with entrypoint/cmd look-up

Open rgdev opened this issue 2 months ago • 1 comments

Pre-requisites

  • [x] I have double-checked my configuration
  • [x] I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
  • [x] I have searched existing issues and could not find a match for this bug
  • [x] I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

I'm encountering an issue on 3.7.2 and latest :

When I deploy an HTTP Template it fails with the error failed to look-up entrypoint/cmd for image \"\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: could not parse reference: which is odd since HTTP Template is supposed to be launched using agent from my understanding.

This line of line also seems to suggest the controller is treating the http template node as a container template : time=2025-10-01T15:40:47.777Z level=DEBUG msg="Executing node with container template" template=maintask component=workflow_worker workflow=httpdebug namespace=workflows nodeName=httpdebug

ConfigMap for the controller :

apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
data:
  mainContainer: |
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: 10m
        memory: 128Mi
      limits:
        memory: 512Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 1000
  executor: |
    image: quay.io/argoproj/argoexec:v3.7.2-nonroot
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: 10m
        memory: 64Mi
      limits:
        cpu: 500m
        memory: 128Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 8737
  # Defaults applied to all tasks (templates)
  workflowDefaults: |
    metadata:
      labels:
        workflows.affluences.com/archive: "true"
    spec:
      serviceAccountName: executor
      podMetadata:
        annotations:
          config.linkerd.io/shutdown-grace-period: "2s"
          config.alpha.linkerd.io/proxy-enable-native-sidecar: true
      ttlStrategy:
        secondsAfterSuccess: 5
      templateDefaults:
        timeout: 3600s
        container:
          securityContext:
            allowPrivilegeEscalation: false
            runAsNonRoot: true
            runAsUser: 1000
            capabilities:
              drop: ["ALL"]
            seccompProfile:
              type: RuntimeDefault
            readOnlyRootFilesystem: true
          volumeMounts:
            - name: tmp
              mountPath: /tmp
        volumes:
          - name: tmp
            emptyDir:
              sizeLimit: 250Mi
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      parallelism: 5

Version(s)

v3.7.2, latest@sha256:88984a28f9d191c1c08420a892ee695b385b9efaa71489229b62144d9aca7efc

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: httpdebug
  namespace: workflows
  labels:
    workflows.argoproj.io/test: "true"
spec:
  entrypoint: maintask
  templates:
    - name: maintask
      http:
        url: "https://www.google.com"

Logs from the workflow controller

time=2025-10-01T15:40:47.757Z level=INFO msg="Processing workflow" component=workflow_worker workflow=httpdebug namespace=workflows Phase="" ResourceVersion=15000100091
time=2025-10-01T15:40:47.757Z level=DEBUG msg="Getting the template by name" workflow=httpdebug namespace=workflows component=workflow_worker name=maintask
time=2025-10-01T15:40:47.757Z level=DEBUG msg="Resolving the template" base="*v1alpha1.Workflow (namespace=workflows,name=httpdebug)" tmpl="*v1alpha1.WorkflowStep (maintask)" component=workflow_worker workflow=httpdebug namespace=workflows
time=2025-10-01T15:40:47.757Z level=DEBUG msg="Getting the template" base="*v1alpha1.Workflow (namespace=workflows,name=httpdebug)" tmpl="*v1alpha1.WorkflowStep (maintask)" component=workflow_worker workflow=httpdebug namespace=workflows
time=2025-10-01T15:40:47.757Z level=DEBUG msg="Getting the template by name" workflow=httpdebug namespace=workflows base="*v1alpha1.Workflow (namespace=workflows,name=httpdebug)" tmpl="*v1alpha1.WorkflowStep (maintask)" name=maintask component=workflow_worker
time=2025-10-01T15:40:47.757Z level=DEBUG msg="Getting the template by name" component=workflow_worker name=maintask workflow=httpdebug namespace=workflows base="*v1alpha1.Workflow (namespace=workflows,name=httpdebug)" tmpl="*v1alpha1.WorkflowStep (maintask)"
time=2025-10-01T15:40:47.757Z level=DEBUG msg="Resolving the template" namespace=workflows base="*v1alpha1.Workflow (namespace=workflows,name=httpdebug)" tmpl="*v1alpha1.WorkflowStep (maintask)" component=workflow_worker workflow=httpdebug
time=2025-10-01T15:40:47.757Z level=DEBUG msg="Getting the template" base="*v1alpha1.Workflow (namespace=workflows,name=httpdebug)" tmpl="*v1alpha1.WorkflowStep (maintask)" component=workflow_worker workflow=httpdebug namespace=workflows
time=2025-10-01T15:40:47.757Z level=DEBUG msg="Getting the template by name" namespace=workflows base="*v1alpha1.Workflow (namespace=workflows,name=httpdebug)" tmpl="*v1alpha1.WorkflowStep (maintask)" name=maintask component=workflow_worker workflow=httpdebug
time=2025-10-01T15:40:47.777Z level=INFO msg="resolved artifact repository" component=workflow_worker workflow=httpdebug namespace=workflows artifactRepositoryRef=default-artifact-repository
time=2025-10-01T15:40:47.777Z level=INFO msg="Task-result reconciliation" workflow=httpdebug namespace=workflows component=workflow_worker numObjs=0
time=2025-10-01T15:40:47.777Z level=DEBUG msg="Skipping artifact GC" component=workflow_worker workflow=httpdebug namespace=workflows
time=2025-10-01T15:40:47.777Z level=INFO msg="updated phase" namespace=workflows component=workflow_worker workflow=httpdebug fromPhase="" toPhase=Running
time=2025-10-01T15:40:47.777Z level=DEBUG msg="Evaluating node" component=workflow_worker nodeName=httpdebug template="*v1alpha1.WorkflowStep (maintask)" boundaryID="" workflow=httpdebug namespace=workflows
time=2025-10-01T15:40:47.777Z level=WARN msg="Node was nil, will be initialized as type Skipped" component=workflow_worker workflow=httpdebug namespace=workflows
time=2025-10-01T15:40:47.777Z level=DEBUG msg="Resolving the template" base="*v1alpha1.Workflow (namespace=workflows,name=httpdebug)" tmpl="*v1alpha1.WorkflowStep (maintask)" component=workflow_worker workflow=httpdebug namespace=workflows
time=2025-10-01T15:40:47.777Z level=DEBUG msg="Getting the template" component=workflow_worker workflow=httpdebug namespace=workflows base="*v1alpha1.Workflow (namespace=workflows,name=httpdebug)" tmpl="*v1alpha1.WorkflowStep (maintask)"
time=2025-10-01T15:40:47.777Z level=DEBUG msg="Getting the template by name" tmpl="*v1alpha1.WorkflowStep (maintask)" component=workflow_worker workflow=httpdebug namespace=workflows base="*v1alpha1.Workflow (namespace=workflows,name=httpdebug)" name=maintask
time=2025-10-01T15:40:47.777Z level=INFO msg="Event occurred" kind=Workflow apiVersion=argoproj.io/v1alpha1 type=Normal reason=WorkflowRunning message="Workflow Running" component=workflow_worker object.name=httpdebug object.namespace=workflows fieldPath=""
time=2025-10-01T15:40:47.777Z level=DEBUG msg="Initializing node" component=workflow_worker workflow=httpdebug namespace=workflows nodeName=httpdebug template="*v1alpha1.WorkflowStep (maintask)" boundaryID=""
time=2025-10-01T15:40:47.777Z level=INFO msg="was unable to obtain node, letting display name to be nodeName" component=workflow_worker boundaryID="" workflow=httpdebug namespace=workflows
time=2025-10-01T15:40:47.777Z level=INFO msg="node initialized" component=workflow_worker workflow=httpdebug namespace=workflows message="" node=httpdebug phase=Pending
time=2025-10-01T15:40:47.777Z level=DEBUG msg="Changing NodeStatus" component=workflow_worker workflow=httpdebug namespace=workflows key=httpdebug status="{ID:httpdebug Name:httpdebug DisplayName:httpdebug Type:Pod TemplateName:maintask TemplateRef:nil TemplateScope:local/httpdebug Phase:Pending BoundaryID: Message: StartedAt:2025-10-01 15:40:47.777544103 +0000 UTC FinishedAt:0001-01-01 00:00:00 +0000 UTC EstimatedDuration:0 Progress: ResourcesDuration: PodIP: Daemoned:<nil> NodeFlag:nil Inputs:nil Outputs:nil Children:[] OutboundNodes:[] HostNodeName: MemoizationStatus:nil SynchronizationStatus:nil TaskResultSynced:0xc000ce1f65}"
time=2025-10-01T15:40:47.777Z level=DEBUG msg="Executing node with container template" template=maintask component=workflow_worker workflow=httpdebug namespace=workflows nodeName=httpdebug
time=2025-10-01T15:40:47.777Z level=DEBUG msg=addArchiveLocation component=workflow_worker workflow=httpdebug namespace=workflows needLocation=true
time=2025-10-01T15:40:47.777Z level=INFO msg=getExecutorLogOpts loglevel=debug namespace=workflows component=workflow_worker workflow=httpdebug
time=2025-10-01T15:40:47.777Z level=INFO msg=getExecutorLogOpts component=workflow_worker workflow=httpdebug namespace=workflows loglevel=debug
time=2025-10-01T15:40:47.777Z level=WARN msg="couldn't get boundaryTemplate" component=workflow_worker workflow=httpdebug namespace=workflows nodeName=httpdebug
time=2025-10-01T15:40:47.778Z level=DEBUG msg="Cache miss" namespace=workflows component=workflow_worker image="" workflow=httpdebug
time=2025-10-01T15:40:47.795Z level=WARN msg="Non-transient error" namespace=workflows error="failed to look-up entrypoint/cmd for image \"\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: could not parse reference: " component=workflow_worker workflow=httpdebug
time=2025-10-01T15:40:47.795Z level=ERROR msg="marking node as error" workflow=httpdebug namespace=workflows error="failed to look-up entrypoint/cmd for image \"\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: could not parse reference: " component=workflow_worker nodeName=httpdebug
time=2025-10-01T15:40:47.795Z level=INFO msg="node phase changed" toPhase=Error node=httpdebug fromPhase=Pending workflow=httpdebug namespace=workflows component=workflow_worker
time=2025-10-01T15:40:47.795Z level=INFO msg="node message changed" component=workflow_worker workflow=httpdebug namespace=workflows node=httpdebug message="failed to look-up entrypoint/cmd for image \"\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: could not parse reference: "
time=2025-10-01T15:40:47.795Z level=INFO msg="node finished" component=workflow_worker workflow=httpdebug namespace=workflows finishedAt=2025-10-01T15:40:47.79568525Z node=httpdebug
time=2025-10-01T15:40:47.795Z level=DEBUG msg="Changing NodeStatus" component=workflow_worker workflow=httpdebug namespace=workflows key=httpdebug status="{ID:httpdebug Name:httpdebug DisplayName:httpdebug Type:Pod TemplateName:maintask TemplateRef:nil TemplateScope:local/httpdebug Phase:Error BoundaryID: Message:failed to look-up entrypoint/cmd for image \"\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: could not parse reference:  StartedAt:2025-10-01 15:40:47.777544103 +0000 UTC FinishedAt:2025-10-01 15:40:47.79568525 +0000 UTC EstimatedDuration:0 Progress: ResourcesDuration: PodIP: Daemoned:<nil> NodeFlag:nil Inputs:nil Outputs:nil Children:[] OutboundNodes:[] HostNodeName: MemoizationStatus:nil SynchronizationStatus:nil TaskResultSynced:0xc001e9a070}"
time=2025-10-01T15:40:47.795Z level=ERROR msg="error in entry template execution" component=workflow_worker workflow=httpdebug namespace=workflows error="failed to look-up entrypoint/cmd for image \"\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: could not parse reference: "
time=2025-10-01T15:40:47.795Z level=WARN msg="Non-transient error" component=workflow_worker workflow=httpdebug namespace=workflows error="failed to look-up entrypoint/cmd for image \"\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: could not parse reference: "
time=2025-10-01T15:40:47.795Z level=DEBUG msg="Task results completion status" component=workflow_worker workflow=httpdebug namespace=workflows status=map[]
time=2025-10-01T15:40:47.795Z level=INFO msg="updated phase" namespace=workflows fromPhase=Running toPhase=Error component=workflow_worker workflow=httpdebug
time=2025-10-01T15:40:47.795Z level=INFO msg="updated message" component=workflow_worker workflow=httpdebug namespace=workflows fromMessage="" toMessage="error in entry template execution: failed to look-up entrypoint/cmd for image \"\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: could not parse reference: "
time=2025-10-01T15:40:47.795Z level=INFO msg="Marking workflow completed" workflow=httpdebug namespace=workflows component=workflow_worker
time=2025-10-01T15:40:47.795Z level=INFO msg="Marking workflow as pending archiving" component=workflow_worker workflow=httpdebug namespace=workflows
time=2025-10-01T15:40:47.795Z level=INFO msg="Event occurred" component=workflow_worker object.name=httpdebug object.namespace=workflows fieldPath="" kind=Workflow apiVersion=argoproj.io/v1alpha1 type=Warning reason=WorkflowFailed message="error in entry template execution: failed to look-up entrypoint/cmd for image \"\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: could not parse reference: "
time=2025-10-01T15:40:47.796Z level=DEBUG msg="Log changes patch" workflow=httpdebug namespace=workflows patch="{\"metadata\":{\"annotations\":{\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"labels\":{\"workflows.company.com/archive\":\"true\",\"workflows.argoproj.io/completed\":\"true\",\"workflows.argoproj.io/phase\":\"Error\",\"workflows.argoproj.io/workflow-archiving-status\":\"Pending\"}},\"spec\":{\"parallelism\":5,\"podMetadata\":{\"annotations\":{\"config.alpha.linkerd.io/proxy-enable-native-sidecar\":\"true\",\"config.linkerd.io/shutdown-grace-period\":\"2s\"}},\"securityContext\":{\"fsGroup\":1000,\"runAsNonRoot\":true,\"runAsUser\":1000,\"seccompProfile\":{\"type\":\"RuntimeDefault\"}},\"serviceAccountName\":\"executor\",\"templateDefaults\":{\"container\":{\"name\":\"\",\"resources\":{},\"securityContext\":{\"allowPrivilegeEscalation\":false,\"capabilities\":{\"drop\":[\"ALL\"]},\"readOnlyRootFilesystem\":true,\"runAsNonRoot\":true,\"runAsUser\":1000,\"seccompProfile\":{\"type\":\"RuntimeDefault\"}},\"volumeMounts\":[{\"mountPath\":\"/tmp\",\"name\":\"tmp\"},{\"mountPath\":\"/secrets/kafka/ca.pem\",\"name\":\"ca-cert\",\"readOnly\":true,\"subPath\":\"certificate\"}]},\"inputs\":{},\"metadata\":{},\"outputs\":{},\"timeout\":\"3600s\",\"volumes\":[{\"emptyDir\":{\"sizeLimit\":\"250Mi\"},\"name\":\"tmp\"},{\"name\":\"ca-cert\",\"secret\":{\"secretName\":\"kafka-ca-certificate\"}}]},\"ttlStrategy\":{\"secondsAfterSuccess\":5}},\"status\":{\"artifactGCStatus\":{\"notSpecified\":true},\"artifactRepositoryRef\":{\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"accessKeySecret\":{\"key\":\"access_key\",\"name\":\"argo-workflows-s3-credentials\"},\"bucket\":\"staging-argo-workflows-artifacts-gra\",\"endpoint\":\"censored\",\"keyFormat\":\"workflows-artifacts/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{workflow.name}}/{{pod.name}}\",\"region\":\"gra\",\"secretKeySecret\":{\"key\":\"secret_key\",\"name\":\"argo-workflows-s3-credentials\"}}},\"default\":true},\"conditions\":[{\"status\":\"True\",\"type\":\"Completed\"}],\"finishedAt\":\"2025-10-01T15:40:47Z\",\"message\":\"error in entry template execution: failed to look-up entrypoint/cmd for image \\\"\\\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: could not parse reference: \",\"nodes\":{\"httpdebug\":{\"displayName\":\"httpdebug\",\"finishedAt\":\"2025-10-01T15:40:47Z\",\"id\":\"httpdebug\",\"message\":\"failed to look-up entrypoint/cmd for image \\\"\\\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: could not parse reference: \",\"name\":\"httpdebug\",\"phase\":\"Error\",\"startedAt\":\"2025-10-01T15:40:47Z\",\"taskResultSynced\":true,\"templateName\":\"maintask\",\"templateScope\":\"local/httpdebug\",\"type\":\"Pod\"}},\"phase\":\"Error\",\"startedAt\":\"2025-10-01T15:40:47Z\"}}" component=workflow_worker
time=2025-10-01T15:40:47.796Z level=DEBUG msg="Changing NodeStatus" status="{ID:httpdebug Name:httpdebug DisplayName:httpdebug Type:Pod TemplateName:maintask TemplateRef:nil TemplateScope:local/httpdebug Phase:Error BoundaryID: Message:failed to look-up entrypoint/cmd for image \"\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: could not parse reference:  StartedAt:2025-10-01 15:40:47.777544103 +0000 UTC FinishedAt:2025-10-01 15:40:47.79568525 +0000 UTC EstimatedDuration:0 Progress:0/1 ResourcesDuration: PodIP: Daemoned:<nil> NodeFlag:nil Inputs:nil Outputs:nil Children:[] OutboundNodes:[] HostNodeName: MemoizationStatus:nil SynchronizationStatus:nil TaskResultSynced:0xc001e9a070}" component=workflow_worker workflow=httpdebug namespace=workflows key=httpdebug
time=2025-10-01T15:40:47.796Z level=INFO msg="Workflow to be dehydrated" namespace=workflows component=workflow_worker workflow=httpdebug "Workflow Size"=2514
time=2025-10-01T15:40:47.887Z level=INFO msg="Workflow update successful" namespace=workflows resourceVersion=15000100725 phase=Error component=workflow_worker workflow=httpdebug
time=2025-10-01T15:40:47.887Z level=INFO msg="Event occurred" apiVersion=argoproj.io/v1alpha1 type=Normal reason=WorkflowNodeRunning message="Running node httpdebug: failed to look-up entrypoint/cmd for image \"\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: could not parse reference: " component=workflow_worker object.name=httpdebug object.namespace=workflows fieldPath="" kind=Workflow
time=2025-10-01T15:40:47.887Z level=INFO msg="Event occurred" type=Warning reason=WorkflowNodeError message="Error node httpdebug: failed to look-up entrypoint/cmd for image \"\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: could not parse reference: " object.name=httpdebug object.namespace=workflows component=workflow_worker fieldPath="" kind=Workflow apiVersion=argoproj.io/v1alpha1
time=2025-10-01T15:40:47.968Z level=INFO msg="archiving workflow" component=archive_worker workflow=httpdebug uid=9415c7b8-d119-40a2-ad45-065a3c512de6 namespace=workflows

Logs from in your workflow's wait container

Pod does not spawn

rgdev avatar Oct 01 '25 16:10 rgdev

This should be a configuration issue. You added container configurations in templateDefaults, which caused the issue.

jswxstw avatar Oct 15 '25 09:10 jswxstw