Emissary does not recognize the Entrypoint in the Docker image

[vitalyrychkov]

Hello,

We have recently upgraded to Argo Workflows 3.3.8 and changed the executor to Emissary. We are using Argo since about two years and built a framework around it. Until now we have used pns executor. After switching to Emissary, all our workflows fail with:
"task errored: container "main" in template

We always provide ENTRYPOINT and CMD in the Dockerfile and override the CMD with args in the contaner spec in the workflow template.

I have seen in this thread (https://github.com/argoproj/argo-workflows/pull/8345#issuecomment-1180223024) that you expect it to work this way:

• If you do not supply command or args for a Container, the defaults defined in the Docker image are used.
• If you supply only args for a Container, the default Entrypoint defined in the Docker image is run with the args that you supplied.
• If you supply a command for a Container, only the supplied command is used. The default EntryPoint and the default Cmd defined in the Docker image are ignored. Your command is run with the args supplied (or no args if none supplied).

However the second point (ENTRYPOINT in the Dockerfile and no command in the Container) does not work. Probably you did not have a test case for this one, I am attaching my simple test case for you then.

Could you please confirm this is the bug which must be fixed?

Originally posted by @vitalyrychkov in https://github.com/argoproj/argo-workflows/issues/8345#issuecomment-1180223024

[vitalyrychkov]

TEST CASE Create a Dockerfile:

FROM debian
ENTRYPOINT ["echo"]
CMD ["DOCKERFILE"]

Build the image, e.g. cmdtest:latest and push to your repo

Create a test workflow to use the built image:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: cmdtest-
  labels:
    workflows.argoproj.io/archive-strategy: "false"
  annotations:
    workflows.argoproj.io/description: |
      This is a test for image command and entrypoint 
spec:
  entrypoint: cmdtest
  templates:
  - name: cmdtest
    container:
      image: <repo>/cmdtest:latest
      #command: ["echo"]
      #args: ["ARGS"]

In our Argo installation with Emissary executor the workflow fails with the mentioned error. If I uncomment the command, then the workflow succeeds.

[vitalyrychkov]

Unfortunately I can not assign a Bug label now, could anyone do it for this issue please?

[terrytangyuan]

@alexec Would you like to take a look?

[alexec]

It maybe the case that they’re using v1 Docker image, not v2. Or it maybe they need to try “:latest”

[vitalyrychkov]

@alexec Thank you for looking into this, I will check with developers about the schema version.

[vitalyrychkov]

The image I have built for the described test case shows the schema version 2: { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 912, "digest": "sha256:2031bebd1be7070e5590d67421c54e49071629ddf17bb2fec0f03bdc1cf76ff7" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 55009886, "digest": "sha256:1339eaac5b67d16d6d9f41fb7a7b96f7cebf3ba4beab36cbb60935aa772af583" } ] }

[vitalyrychkov]

@alexec I understand that you are busy with lots of other issues, just note that the test case above takes not more than 10-15 minutes to perform. In the version 3.3.8 we can still use the PNS executor, but would appreciate if this issue is solved before Emissary will be the only supported executor.

[leoncider]

This is a big problem for us as well

[alexec]

Can you repro on latest?

[vitalii-cidersecurity]

Can you repro on latest?

Tried on https://github.com/argoproj/argo-workflows/releases/tag/v3.3.8. Received the same issue.

[alexec]

Sorry. By “latest” I mean ”:latest” image which is built from master branch.

[vitalyrychkov]

Hi Alex, thank you very much for the fix!

With the "latest" tag deployed, I have repeated my test case:

I assume that Emissary is the only supported executor in the latest build and does not recognize the executor-related keywords: containerRuntimeExecutor and containerRuntimeExecutors. I had to remove them from the configmap for the workflow controller to start successfully.

Then I have submitted my test command and got the MESSAGE (found it only in the Argo GUI): failed to look-up entrypoint/cmd for image "<image_registry_host>/vitaly/cmdtest:latest", you must either explicitly specify the command, or list the image's command in the index: https://argoproj.github.io/argo-workflows/workflow-executors/#emissary-emissary: Get "https://<image_registry_host>/v2/": x509: certificate signed by unknown authority

We have already had the internal root CA mounted as a volume to the Argo server container, but until now we did not need it for the controller container.

Added the internal root CA as a volume mount to the controller container:

volumeMounts: 
- name: myca
  mountPath: /etc/ssl/certs/ca.crt
  subPath: ca.crt #
volumes: 
- name: myca
  secret:
    secretName: myca

Now the cmdtest pod starts and runs to success, here is the init log:

	time="2022-08-04T11:52:57.119Z" level=info msg="Starting Workflow Executor" version=untagged
	time="2022-08-04T11:52:57.123Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
	time="2022-08-04T11:52:57.123Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=default podName=cmdtest-24zk4 template="{\"name\":\"cmdtest\",\"inputs\":{},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"<image_registry_host>/vitaly/cmdtest:latest\",\"resources\":{}}}" version="&Version{Version:untagged,BuildDate:2022-08-03T13:59:35Z,GitCommit:06b0a8cce637db1adae0bae91670e002cfd0ae4d,GitTag:untagged,GitTreeState:clean,GoVersion:go1.18.5,Compiler:gc,Platform:linux/amd64,}"
	time="2022-08-04T11:52:58.269Z" level=info msg="Start loading input artifacts..."
	time="2022-08-04T11:52:58.269Z" level=info msg="Alloc=5814 TotalAlloc=11560 Sys=24274 NumGC=4 Goroutines=2"

So, as far as I understand, the issue was the following: unlike PNS, Emissary tries to pull the image manifest from the image repository. In our case our development image registry uses a self-signed certificate and Argo workflow controller could not create TLS connection (which is properly communicated as a message in the latest build). Mounting the CA to the container solves the issue. Am I right?

Would you recommend to keep mounting the CA as a volume to both Argo server and workflow controller as a volume, or is there a better approach?

[alexec]

Thank you for your detailed explanation. I think other uses will find this helpful. I think your approach is fine.

[vitalyrychkov]

Thank you Alex, we really love Argo.

[alexec]

I think this issue is resolved now.

[laymain]

Hi, does it mean that the command arg will no longer be required for emissary executor in v3.4?
If so, shouldn't the documentation be updated?

https://github.com/argoproj/argo-workflows/blob/v3.4.0-rc4/docs/workflow-executors.md#:~:text=command%20must%20be%20specified%20for%20containers.

[Timvissers]

Hi, does it mean that the command arg will no longer be required for emissary executor in v3.4? If so, shouldn't the documentation be updated?

same question here. From testing it seems it's still necessary.. (which contradicts the description here iiuc)

[SiddharthPant]

Hi, does it mean that the command arg will no longer be required for emissary executor in v3.4? If so, shouldn't the documentation be updated?

same question here. From testing it seems it's still necessary.. (which contradicts the description here iiuc)

From what I am seeing It's not necessary or rather it's only necessary if you have not configured the certificate to your docker image repo as shown above(by post from @vitalyrychkov) in your workflow.