Artifact upload to GCS with S3 protocol appends chunk-signature

[vilmosnagy]

Pre-requisites

• [X] I have double-checked my configuration
• [X] I can confirm the issue exists when I tested with :latest
• [X] I have searched existing issues and could not find a match for this bug
• [ ] I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

Hi,

We've identified an issue after upgrading from Argo v3.3.8 to the latest version, v3.5.4. Our workflows, previously running smoothly in the Google Cloud environment, are now encountering problems.

Upon investigation, we traced the issue back to our use of Google Cloud Storage as an object storage for artifacts with S3 interoperability enabled, following the example provided here. This S3 interoperability is vital for us as we leverage these workflows across different cloud providers.

Post-upgrade, an unexpected prefix and suffix have been introduced at the beginning and end of our artifacts. This addition, present in both binary and text files, is disrupting our previously functional workflows.

For illustration, consider the following example where we anticipate the output to be:

random text

Unfortunately, when uploading to Google Cloud, the actual content of the output includes the following:

c;chunk-signature=b1728939f1675f68ec1d6f99243a7505c734d32a1237cd8eecb153ef27852441
random text

0;chunk-signature=ad8d9b929ab29f4a556240663b8024908f3ee87779bc16e5d25eb100c1585b20

Interestingly, when we run the same workflow and upload the artifact to a Minio endpoint, everything works as expected.

I believe the issue might be related to something in the minio-go package, particularly around this section. However, my familiarity with Go is limited, and I may not be able to address this effectively.

Additionally, it's worth considering this issue on the minio-go repository, as it might be related to the problem we're facing.

One potential solution could be adding the DisableContentSha256 option at this location. It seems that this option was not present in the [email protected] package, which is utilized by Argo v3.3.8.

I'm uncertain about the effectiveness of this approach and would appreciate your insights or guidance on whether including the DisableContentSha256 option could be a valid solution.

Thank you for your time and consideration.

Version

v3.5.4

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: test-wf-for-gcs-storage
spec:
  serviceAccountName: data-manager-workflows
  entrypoint: create-test-file
  templates:
    - name: create-test-file
      script:
        image: alpine
        command: ["sh", "-e"]
        source: |
          touch  /tmp/test.txt
          echo "random text" > /tmp/test.txt
        env:
      outputs:
        artifacts:
          - name: test-file
            path: /tmp/test.txt
            archive:
              none: { }
            s3:
              endpoint: storage.googleapis.com
              bucket: io-realcity-apps-szombathely-test-data
              key: vilmos-test-argo-update/{{workflow.name}}.txt
              insecure: true
              accessKeySecret:
                name: s3-credentials-argo-update-test
                key: accesskey
              secretKeySecret:
                name: s3-credentials-argo-update-test
                key: secretkey

Logs from the workflow controller

time="2024-01-15T14:55:55.027Z" level=info msg="Processing workflow" Phase= ResourceVersion=548730948 namespace=szombathely workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:55:55.031Z" level=info msg="Task-result reconciliation" namespace=szombathely numObjs=0 workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:55:55.031Z" level=info msg="Updated phase  -> Running" namespace=szombathely workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:55:55.031Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=szombathely workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:55:55.031Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=szombathely workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:55:55.031Z" level=info msg="Pod node test-wf-for-gcs-storage62jk6 initialized Pending" namespace=szombathely workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:55:55.050Z" level=info msg="Created pod: test-wf-for-gcs-storage62jk6 (test-wf-for-gcs-storage62jk6)" namespace=szombathely workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:55:55.050Z" level=info msg="TaskSet Reconciliation" namespace=szombathely workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:55:55.050Z" level=info msg=reconcileAgentPod namespace=szombathely workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:55:55.060Z" level=info msg="Workflow update successful" namespace=szombathely phase=Running resourceVersion=548730951 workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:56:05.052Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=548730951 namespace=szombathely workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:56:05.052Z" level=info msg="Task-result reconciliation" namespace=szombathely numObjs=1 workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:56:05.052Z" level=info msg="task-result changed" namespace=szombathely nodeID=test-wf-for-gcs-storage62jk6 workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:56:05.052Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/" namespace=szombathely workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:56:05.052Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/" namespace=szombathely workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:56:05.052Z" level=info msg="node changed" namespace=szombathely new.message= new.phase=Succeeded new.progress=0/1 nodeID=test-wf-for-gcs-storage62jk6 old.message= old.phase=Pending old.progress=0/1 workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:56:05.053Z" level=info msg="TaskSet Reconciliation" namespace=szombathely workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:56:05.053Z" level=info msg=reconcileAgentPod namespace=szombathely workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:56:05.053Z" level=info msg="Updated phase Running -> Succeeded" namespace=szombathely workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:56:05.053Z" level=info msg="Marking workflow completed" namespace=szombathely workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:56:05.053Z" level=info msg="Marking workflow as pending archiving" namespace=szombathely workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:56:05.059Z" level=info msg="cleaning up pod" action=deletePod key=szombathely/test-wf-for-gcs-storage62jk6-1340600742-agent/deletePod
time="2024-01-15T14:56:05.061Z" level=info msg="Workflow update successful" namespace=szombathely phase=Succeeded resourceVersion=548731073 workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:56:05.091Z" level=info msg="archiving workflow" namespace=szombathely uid=c92fd8c7-bf6d-4a35-86ce-844b64e01f70 workflow=test-wf-for-gcs-storage62jk6
time="2024-01-15T14:56:05.096Z" level=info msg="cleaning up pod" action=labelPodCompleted key=szombathely/test-wf-for-gcs-storage62jk6/labelPodCompleted

Logs from in your workflow's wait container

time="2024-01-15T14:55:56.867Z" level=info msg="Starting Workflow Executor" version=v3.5.4
time="2024-01-15T14:55:56.872Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2024-01-15T14:55:56.873Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=szombathely podName=test-wf-for-gcs-storage62jk6 templateName=create-test-file version="&Version{Version:v3.5.4,BuildDate:2024-01-14T05:30:37Z,GitCommit:960af331a8c0a3f2e263c8b90f1daf4303816ba8,GitTag:v3.5.4,GitTreeState:clean,GoVersion:go1.21.6,Compiler:gc,Platform:linux/amd64,}"
time="2024-01-15T14:55:56.890Z" level=info msg="Starting deadline monitor"
time="2024-01-15T14:55:59.891Z" level=info msg="Main container completed" error="<nil>"
time="2024-01-15T14:55:59.891Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2024-01-15T14:55:59.891Z" level=info msg="No output parameters"
time="2024-01-15T14:55:59.891Z" level=info msg="Saving output artifacts"
time="2024-01-15T14:55:59.891Z" level=info msg="Staging artifact: test-file"
time="2024-01-15T14:55:59.891Z" level=info msg="Copying /tmp/test.txt from container base image layer to /tmp/argo/outputs/artifacts/test-file.tgz"
time="2024-01-15T14:55:59.891Z" level=info msg="/var/run/argo/outputs/artifacts/tmp/test.txt.tgz -> /tmp/argo/outputs/artifacts/test-file.tgz"
time="2024-01-15T14:55:59.891Z" level=info msg="Untaring /tmp/argo/outputs/artifacts/test-file.tgz archive before upload"
time="2024-01-15T14:55:59.896Z" level=info msg="S3 Save path: /tmp/argo/outputs/artifacts/test.txt, key: vilmos-test-argo-update/test-wf-for-gcs-storage62jk6.txt"
time="2024-01-15T14:55:59.896Z" level=info msg="Creating minio client using static credentials" endpoint=storage.googleapis.com
time="2024-01-15T14:55:59.896Z" level=info msg="Saving file to s3" bucket=io-realcity-apps-szombathely-test-data endpoint=storage.googleapis.com key=vilmos-test-argo-update/test-wf-for-gcs-storage62jk6.txt path=/tmp/argo/outputs/artifacts/test.txt
time="2024-01-15T14:56:00.395Z" level=info msg="Save artifact" artifactName=test-file duration=498.741692ms error="<nil>" key=vilmos-test-argo-update/test-wf-for-gcs-storage62jk6.txt
time="2024-01-15T14:56:00.395Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/artifacts/test.txt
time="2024-01-15T14:56:00.395Z" level=info msg="Successfully saved file: /tmp/argo/outputs/artifacts/test.txt"
time="2024-01-15T14:56:00.395Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: test-wf-for-gcs-storage62jk6/test-wf-for-gcs-storage62jk6/main.log"
time="2024-01-15T14:56:00.395Z" level=info msg="Creating minio client using static credentials" endpoint=argo-s3.argo-system
time="2024-01-15T14:56:00.395Z" level=info msg="Saving file to s3" bucket=argo endpoint=argo-s3.argo-system key=test-wf-for-gcs-storage62jk6/test-wf-for-gcs-storage62jk6/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-01-15T14:56:00.599Z" level=info msg="Save artifact" artifactName=main-logs duration=204.05243ms error="<nil>" key=test-wf-for-gcs-storage62jk6/test-wf-for-gcs-storage62jk6/main.log
time="2024-01-15T14:56:00.599Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-01-15T14:56:00.599Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-01-15T14:56:00.613Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/" error="workflowtaskresults.argoproj.io \"test-wf-for-gcs-storage62jk6\" is forbidden: User \"system:serviceaccount:szombathely:data-manager-workflows\" cannot patch resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"szombathely\""
time="2024-01-15T14:56:00.633Z" level=info msg="Alloc=12972 TotalAlloc=19100 Sys=33637 NumGC=4 Goroutines=12"
time="2024-01-15T14:56:00.635Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/" error="workflowtaskresults.argoproj.io \"test-wf-for-gcs-storage62jk6\" is forbidden: User \"system:serviceaccount:szombathely:data-manager-workflows\" cannot patch resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"szombathely\""

[agilgur5]

One potential solution could be adding the DisableContentSha256 option at this location.

Exposing this option in artifact config could make sense

Unfortunately, when uploading to Google Cloud, the actual content of the output includes the following:

c;chunk-signature=b1728939f1675f68ec1d6f99243a7505c734d32a1237cd8eecb153ef27852441
random text

0;chunk-signature=ad8d9b929ab29f4a556240663b8024908f3ee87779bc16e5d25eb100c1585b20

Interestingly, when we run the same workflow and upload the artifact to a Minio endpoint, everything works as expected.

Is there a way for you to configure GCS to accept chunk signatures? Since that appears to be supported by MinIO etc

[vilmosnagy]

Okay, the comment from here suggests that removing the insecure: true part will remove the signature from the upload.

At first glance this seems to solve the issue for me - let me run a double check in the coming days / weeks, I'll get back to this issue in a few weeks.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity and needs more information. It will be closed if no further activity occurs.

[github-actions[bot]]

This issue has been closed due to inactivity and lack of information. If you still encounter this issue, please add the requested information and re-open.

[vilmosnagy]

At first glance this seems to solve the issue for me - let me run a double check in the coming days / weeks, I'll get back to this issue in a few weeks.

for future reference: this solved our issue with minio.