ormb push error when pushing model in to harbor registry

[hemantha-kumara]

/kind bug

What happened: Trying to push the model into the harbor registry. harbor login is successful but ormb push is failing with the below error

sh-4.2# ormb push  harbor.dsml.net/kubeflowprofiles/xgboostsample:v1
The push refers to repository [harbor.dsm.net/kubeflowprofiles/xgboostsample]
ref:       harbor.dsml.net/kubeflowprofiles/xgboostsample:v1
digest:    b57ea86dceff41196d633005bbaeb70af71ad2a7b3e8a8f490ad06757d1436fa
size:      905 B
format:    XGBoost
Error: failed to authorize: failed to fetch oauth token: unable to decode token response: invalid character '<' looking for beginning of value

What you expected to happen:

ormb push should be successible

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

PANI[2021-01-05T14:27:43Z] Failed to run the  command                    error="failed to authorize: failed to fetch oauth token: unable to decode token response: invalid character '<' looking for beginning of value"
panic: (*logrus.Entry) (0xb9acc0,0xc000122150)

goroutine 1 [running]:
github.com/sirupsen/logrus.Entry.log(0xc000122000, 0xc00001e9c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/bytedance/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:239 +0x2da
github.com/sirupsen/logrus.(*Entry).Log(0xc000122070, 0x0, 0xc00059ff08, 0x1, 0x1)
        /Users/bytedance/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:268 +0xeb
github.com/sirupsen/logrus.(*Entry).Logln(0xc000122070, 0x0, 0xc00059ff58, 0x1, 0x1)
        /Users/bytedance/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:359 +0xd1
github.com/sirupsen/logrus.(*Entry).Panicln(...)
        /Users/bytedance/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:397
github.com/kleveross/ormb/cmd/ormb/cmd.Execute()
        /Users/bytedance/go/src/github.com/kleveross/ormb/cmd/ormb/cmd/root.go:45 +0xcb
main.main()
        /Users/bytedance/go/src/github.com/kleveross/ormb/cmd/ormb/main.go:23 +0x20

sh-4.2# ormb version
Version: 0.0.9
Git SHA: 92e4a8dd3a61d0a3d5c6071217252d302daeafba
Repo Root: [email protected]:simon-cj/ormb.git
Go Version: go1.14.8
Go OS/Arch: linux/amd64

[gaocegege]

/assign @simon-cj @FogDong

[gaocegege]

Can you please tell us What the Harbor version you are using?

[hemantha-kumara]

Can you please tell us What the Harbor version you are using?

Harbor Version v1.8.0

[FogDong]

Do you have a project named KubeFlowProfiles in your harbor? @hemantha-kumara Right now we do not support create the project automatically so the project must be created first.

[hemantha-kumara]

Do you have a project named KubeFlowProfiles in your harbor? @hemantha-kumara Right now we do not support create the project automatically so the project must be created first.

Yes, the project kubeflowprofiles exists

[hemantha-kumara]

The error is Error: failed to authorize: failed to fetch oauth token . In our case harbor authentication is enabled using OIDC(Keycloak). This might give you some clue !

[simon-cj]

For OIDC((Keycloak), you can test that does docker login and docker push work? I doubt the issue is service token.

[hemantha-kumara]

For OIDC((Keycloak), you can test that does docker login and docker push work? I doubt the issue is service token.

Yes, docker login and docker push work fine. I can push the docker image into the harbor.

[simon-cj]

Can you upgrade Harbor to > v2.1.0, because we develop ormb with harbor community at harbor v2.1.0, I am not sure how it work at v2.1.0 below. I test harbor at v2.0.0, I am sure it will not show the metadata in harbor UI. (The following image show the metadata at harbor v2.1.0) as for the error message, it looks like service token issue, you can test it use harbor service token.

[hemantha-kumara]

as for the error message, it looks like service token issue, you can test it use harbor service token.

Is there any example on how to use harbor service token ?

[simon-cj]

as for the error message, it looks like service token issue, you can test it use harbor service token.

Is there any example on how to use harbor service token ?

After install harbor, it will use default service token, it is define at https://github.com/goharbor/harbor-helm/blob/master/templates/nginx/configmap-http.yaml#L133, it is redirect to harbor-harbor-core componentt by nginx.

if you install harbor in k8s, you can execute kubectl edit configmap harbor-harbor-nginx to set it to default service token. if not install harbor in k8s, you can set nginx config(nginx.conf) too.

[simon-cj]

as for the error message, it looks like service token issue, you can test it use harbor service token.

Is there any example on how to use harbor service token ?

the default config like this in k8s:

   location /service/ {
          proxy_pass http://core/service/;
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

          # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
          proxy_set_header X-Forwarded-Proto $scheme;

          proxy_buffering off;
          proxy_request_buffering off;
        }

[hemantha-kumara]

I tried oras push, even oras push is failing for the same reason. I have created an issue on oras repo also https://github.com/deislabs/oras/issues/205.

here is the log for your reference

sh-4.2# oras push harbor.dsml-csfp-2.ccsp.dyn.nesc.nokia.net/kubeflowprofiles/hello-artifact:v1 --manifest-config /dev/null:application/vnd.acme.rocket.config ./artifact.txt --insecure
Uploading a948904f2f0f artifact.txt
Error: failed to authorize: failed to fetch oauth token: unable to decode token response: invalid character '<' looking for beginning of value