Harbor-registry which set store data on S3 seems not works with service account (IAM role) on EKS

[phuongleeo]

errors:

time="2020-09-10T01:19:26.639004243Z" level=debug msg="authorizing request" go.version=go1.14.5 http.request.host=harbor-domain-xxx http.request.id=4784f85b-81e6-436a-ad15-1d80bd592f13 http.request.method=GET http.request.remoteaddr=10.51.5.130 http.request.uri="/v2/" http.request.useragent="docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(darwin\))" 
10.51.3.247 - - [10/Sep/2020:01:19:26 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.17+"
time="2020-09-10T01:19:26.735805324Z" level=info msg="authorized request" go.version=go1.14.5 http.request.host=harbor-domain-xxx http.request.id=4784f85b-81e6-436a-ad15-1d80bd592f13 http.request.method=GET http.request.remoteaddr=10.51.5.130 http.request.uri="/v2/" http.request.useragent="docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(darwin\))" 
time="2020-09-10T01:19:26.735893065Z" level=info msg="response completed" go.version=go1.14.5 http.request.host=harbor-domain-xxx http.request.id=4784f85b-81e6-436a-ad15-1d80bd592f13 http.request.method=GET http.request.remoteaddr=10.51.5.130 http.request.uri="/v2/" http.request.useragent="docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=98.071917ms http.response.status=200 http.response.written=2 
10.51.3.79 - - [10/Sep/2020:01:19:26 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \\(darwin\\))"
time="2020-09-10T01:19:48.657499925Z" level=debug msg="authorizing request" go.version=go1.14.5 http.request.host=harbor-domain-xxx http.request.id=bad978f9-71cd-4174-ba4a-8ce116c42568 http.request.method=HEAD http.request.remoteaddr=10.51.5.130 http.request.uri="/v2/library/alpine/blobs/sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9" http.request.useragent="docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(darwin\))" vars.digest="sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9" vars.name="library/alpine" 
time="2020-09-10T01:19:48.75854612Z" level=info msg="authorized request" go.version=go1.14.5 http.request.host=harbor-domain-xxx http.request.id=bad978f9-71cd-4174-ba4a-8ce116c42568 http.request.method=HEAD http.request.remoteaddr=10.51.5.130 http.request.uri="/v2/library/alpine/blobs/sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9" http.request.useragent="docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(darwin\))" vars.digest="sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9" vars.name="library/alpine" 
time="2020-09-10T01:19:48.758673571Z" level=debug msg=GetBlob auth.user.name="harbor_registry_user" go.version=go1.14.5 http.request.host=harbor-domain-xxx http.request.id=bad978f9-71cd-4174-ba4a-8ce116c42568 http.request.method=HEAD http.request.remoteaddr=10.51.5.130 http.request.uri="/v2/library/alpine/blobs/sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9" http.request.useragent="docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(darwin\))" vars.digest="sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9" vars.name="library/alpine" 
time="2020-09-10T01:19:48.76247003Z" level=info msg="redis: connect harbor-harbor-redis:6379" go.version=go1.14.5 instance.id=49352e1f-36e6-40e8-90a6-49489e8930a1 redis.connect.duration=3.746778ms service=registry version=v2.7.1.m 
time="2020-09-10T01:20:09.029240861Z" level=debug msg="s3aws.GetContent("/docker/registry/v2/repositories/library/alpine/_layers/sha256/c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9/link")" auth.user.name="harbor_registry_user" go.version=go1.14.5 http.request.host=harbor-domain-xxx http.request.id=bad978f9-71cd-4174-ba4a-8ce116c42568 http.request.method=HEAD http.request.remoteaddr=10.51.5.130 http.request.uri="/v2/library/alpine/blobs/sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9" http.request.useragent="docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(darwin\))" trace.duration=20.265619657s trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=fb0b1a11-c26c-40b2-ac53-ea80eee835be trace.line=95 vars.digest="sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9" vars.name="library/alpine" 
10.51.3.79 - - [10/Sep/2020:01:19:48 +0000] "HEAD /v2/library/alpine/blobs/sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9 HTTP/1.1" 500 104 "" "docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \\(darwin\\))"
time="2020-09-10T01:20:09.029512644Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code=unknown err.detail="s3aws: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors" err.message="unknown error" go.version=go1.14.5 http.request.host=harbor-domain-xxx http.request.id=bad978f9-71cd-4174-ba4a-8ce116c42568 http.request.method=HEAD http.request.remoteaddr=10.51.5.130 http.request.uri="/v2/library/alpine/blobs/sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9" http.request.useragent="docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=20.373355247s http.response.status=500 http.response.written=104 vars.digest="sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9" vars.name="library/alpine" 
time="2020-09-10T01:20:10.266202433Z" level=debug msg="authorizing request" go.version=go1.14.5 http.request.host=harbor-domain-xxx http.request.id=c984aee0-8792-416c-b0ce-e2b116d2ad92 http.request.method=POST http.request.remoteaddr=10.51.3.247 http.request.uri="/v2/library/alpine/blobs/uploads/" http.request.useragent="docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(darwin\))" vars.name="library/alpine" 
time="2020-09-10T01:20:10.361981022Z" level=info msg="authorized request" go.version=go1.14.5 http.request.host=harbor-domain-xxx http.request.id=c984aee0-8792-416c-b0ce-e2b116d2ad92 http.request.method=POST http.request.remoteaddr=10.51.3.247 http.request.uri="/v2/library/alpine/blobs/uploads/" http.request.useragent="docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(darwin\))" vars.name="library/alpine" 
time="2020-09-10T01:20:10.362089723Z" level=debug msg="(*linkedBlobStore).Writer" auth.user.name="harbor_registry_user" go.version=go1.14.5 http.request.host=harbor-domain-xxx http.request.id=c984aee0-8792-416c-b0ce-e2b116d2ad92 http.request.method=POST http.request.remoteaddr=10.51.3.247 http.request.uri="/v2/library/alpine/blobs/uploads/" http.request.useragent="docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(darwin\))" vars.name="library/alpine" #

the service account which already has permission to s3 bucket worked normally in the harbor-chartmuseum service, I could put the chart via helm push. However, the harbor-registry service can not ( docker push always failed), It works only once I patch the harbor-registry config map that included accesskey/secretkey.

the following configmap which did not work:

apiVersion: v1
data:
  config.yml: |
    version: 0.1
    log:
      level: info
      fields:
        service: registry
    storage:
      s3:
        region: eu-central-1
        regionendpoint: https://s3.eu-central-1.amazonaws.com
        v4auth: true
        bucket: my-test-bucket
        rootdirectory: harbor
      cache:
        layerinfo: redis
      maintenance:
        uploadpurging:
          enabled: false
      delete:
        enabled: true
      redirect:
        disable: true
    redis:
      addr: "harbor-harbor-redis:6379"
      db: 2
    http:
      addr: :5000
      relativeurls: true
      # set via environment variable
      # secret: placeholder
      debug:
        addr: localhost:5001
    auth:
      htpasswd:
        realm: harbor-registry-basic-realm
        path: /etc/registry/passwd
    validation:
      disabled: true
  ctl-config.yml: |
    ---
    protocol: "http"
    port: 8080
    log_level: info
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: harbor
    meta.helm.sh/release-namespace: bootstrap
  labels:
    app: harbor
    app.kubernetes.io/managed-by: Helm
    chart: harbor
    heritage: Helm
    release: harbor
  name: harbor-harbor-registry
  namespace: bootstrap

[kschu91]

Having the same issue. Seems like that the harbor registry does not support sts:AssumeRoleWithWebIdentity. Would love to see that as well.

[skaymakca]

The issue on the harbor repo seems closed, but the problem remains. The service account method to give S3 access still doesn't work in EKS and access keys still need to be created.

[sydorovdmytro]

Any updates here? Does anyone work on this issue?

[rokkarinn]

I am also curious if this will be fixed?

[darend]

It appears to be an issue in the distribution engine, see this issue https://github.com/distribution/distribution/issues/3275#issuecomment-907399342

It has been address in the main branch https://github.com/distribution/distribution/issues/3097 but they have not performed a release since 2019

[darend]

I submitted a PR to update distribution to a version that supports AWS AssumeRoleWithWebIdentity https://github.com/goharbor/harbor/pull/16190

[joao-dantas]

Also looking forward to updates here.

[ghost]

is there any updates on this?

[smauermann]

It seems that this is not fixed in the upstream dependency distribution :/ https://github.com/distribution/distribution/issues/3275#issuecomment-1163439385

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[bootc]

Just keeping this fresh. It's still a problem.

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[bootc]

This needs to stay open.