fleet apply with private helm chart registry works with 0.11.7 but not 0.12.2

[chrisfarnham]

Is there an existing issue for this?

• [x] I have searched the existing issues

Current Behavior

Hello, I've reviewed the release notes and feel I must have missed something.

In our CI we configure authentication via a helm registry login command, like so:

echo '${{ secrets.HARBOR_PASSWORD }}' | helm registry login --username '${{ secrets.HARBOR_USERNAME }}' --password-stdin docker.private-repo.com

And then we do a fleet apply to fleet bundles that reference charts in docker.private-repo.com.

This all works when the fleet CLI is 0.11.7. But does not work with fleet CLI 0.12.2. With 0.12.2 the registry responds like so:

unexpected status code 401: unauthorized: unauthorized to access repository: artisight-ecr/charts/artisight-data-service, action: pull: unauthorized to access repository: [redacted repo path] action: pull"

Expected Behavior

No response

Steps To Reproduce

No response

Environment

- Architecture:
- Fleet Version:
- Cluster:
  - Provider:
  - Options:
  - Kubernetes Version:

Logs

Anything else?

No response

[manno]

Can you provide more information on how you specify the helm chart in fleet.yaml? Are you just using 'chart:' to point to the tarball?

We will have to investigate, but 0.12 switches the Helm SDK from 3.16 to 3.17.

Does it work if you use --password-file? (https://fleet.rancher.io/cli/fleet-cli/fleet_apply)

[chrisfarnham]

Here is a slightly redacted snippet of our fleet.yaml

helm:
  releaseName: x-service
  timeoutSeconds: 600
  waitForJobs: true
  chart: "oci://docker.xxxx.com/xxx-ecr/charts/x-service"
  version: '0.1.x-x'
  values: {}
  valuesFiles: []
  valuesFrom:
    - configMapKeyRef:
        name: x-service-values
        namespace: default
        key: values.yaml

[chrisfarnham]

I'll have to try the --password-file approach tomorrow morning.

Oddly, this seems to work in our local development environments on our Mac laptops. But the CI failures are in our Linux based CI environment.

[chrisfarnham]

To test in an environment I have more control over I used the following Dockerfile to create an image configured similar to our CI image:

FROM ubuntu:latest
RUN apt-get update -y
RUN apt-get upgrade -y
RUN apt-get install -y curl openssl
RUN curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl" \
    && chmod +x ./kubectl \
    && mv ./kubectl /usr/local/bin/kubectl \
    && curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 \
    && chmod +x get_helm.sh && ./get_helm.sh
RUN FLEET_VERSION=0.12.2 && \
    ARCH=$(uname -m) && \
    if [ "$ARCH" = "x86_64" ]; then \
        ARCH="amd64"; \
    elif [ "$ARCH" = "aarch64" ]; then \
        ARCH="arm64"; \
    else \
        echo "Unsupported architecture: $ARCH"; \
        exit 1; \
    fi && \
    echo "Installing Fleet v${FLEET_VERSION} for architecture: $ARCH" && \
    mkdir -p /usr/local/bin && \
    curl -sL -o /usr/local/bin/fleet \
        "https://github.com/rancher/fleet/releases/download/v${FLEET_VERSION}/fleet-linux-$ARCH" && \
    chmod +x /usr/local/bin/fleet

I then tried different iterations of helm login then fleet apply --output test.bundle. When I had FLEET_VERSION as 0.11.7 it worked as expected. But with 0.12.2 it does not work. I tried different iterations using --password-file and --oci-password-file and it didn't work.

# How I built the docker images and mounted it in the root directory of my fleet managed project
#  docker build -t test .
#  docker run --rm -it --volume .:/src test


## Inside the running container

# the commands that I ran inside the docker container. I tried different iterations of --password-file and --oci-password-file
echo "XXXXXpasswordXXXX" | helm registry login --username 'XXXXusernamXXXX' --password-stdin docker.corpname.com
fleet apply dev /src/fleet/user-service/ --output test.bundle



# Here's where it worked
root@fa9ffe8fa515:/# echo "XXXXXpasswordXXXX" | helm registry login --username 'XXXXusernamXXXX' --password-stdin docker.corpname.com
Login Succeeded
root@fa9ffe8fa515:/# fleet apply dev /src/fleet/user-service/ --output test.bundle --debug
root@fa9ffe8fa515:/# fleet --version
fleet version v0.11.7 (4809cf03c8926d834067bf1111ea24df6b875c32)

# Here's where it failed
root@f778acb7eb19:/# echo "XXXXXpasswordXXXX" | helm registry login --username 'XXXXusernamXXXX' --password-stdin docker.corpname.com
root@f778acb7eb19:/# helm pull oci://docker.corpname.com/corpname-ecr/charts/corpname-service
Pulled: docker.corpname.com/corpname-ecr/charts/corpname-service:0.1.15
Digest: sha256:7bedcc0584b44c372f4381d2e8256d2e0b3c695a2d8c43f58cd67e2486ea6492
root@f778acb7eb19:/# fleet apply dev /src/fleet/service --output test.bundle
FATA[0000] failed to process bundle: loading directory .chart/f0b3e76060248f917a3828eb6eca88e8a51f3933e852fbc251daf5d03fa7f984, src/fleet/service/service: helm chart download: GET "https://docker.corpname.com/v2/corpname-ecr/charts/corpname-service/tags/list": unexpected status code 401: unauthorized: unauthorized to access repository: corpname-ecr/charts/corpname-service, action: pull: unauthorized to access repository: corpname-ecr/charts/corpname-service, action: pull

[chrisfarnham]

Here's the bundle the above is referencing, which I've redacted slightly

defaultNamespace: service
namespaceLabels:
  local-secrets: "true"
dependsOn:
  - selector:
      matchLabels:
        bundle: kafka
  - selector:
      matchLabels:
        bundle: cloudnative-pg
helm:
  releaseName: service
  timeoutSeconds: 600
  waitForJobs: true
  chart: "oci://docker.corpname.com/corpname-ecr/charts/corpname-service"
  version: '0.1.x-x'
  values: {}
  valuesFiles: []
  valuesFrom:
    - configMapKeyRef:
        name: service-values
        namespace: default
        key: values.yaml

diff:
  comparePatches:
  - apiVersion: postgresql.cnpg.io/v1
    kind: Cluster
    name: service-cluster
    namespace: service
    jsonPointers: # jsonPointers allows to ignore diffs at certain json path
      - "/spec/managed/roles"
      - "/spec/postgresql/pg_hba"
      - "/spec/postgresql/pg_ident"

targetCustomizations:
  - name: '0.1.x-x'
    helm:
      version: '0.1.x-x'
    clusterSelector:
      matchLabels:
        corpname.com/service: '0.1.x-x'
  - name: '0.1.x'
    helm:
      version: '0.1.x'
    clusterSelector:
      matchLabels:
        corpname.com/service: '0.1.x'

[weyfonk]

This is unsupported, and was probably broken by this commit in Helm v3.17.0, with v0.11.7 using v3.16.

[chrisfarnham]

FWIW - I believe that what we have been experiencing here is a variant of this issue https://github.com/rancher/fleet/issues/3560

We'll be watching that issue closely and add more information if necessary.