database migration fails when using `enableMigrateHelmHook: true`

[aureq]

I'm trying to upgrade my Harbor registry to 2.5.0 from 2.4.1 (see details below). Since the doc doesn't explains how the db migration is performed when enableMigrateHelmHook is set to false, I decided I would enable the option.

As it turns out, the Job fails (see error below) complaining it can't connect to the database.

I opened a shell into the registry to use psql as a way to check for the db credentials.

1. I ran psql postgres -h 10.42.2.122 -d registry and pasted the password and it worked fine.
2. I ran psql postgres -h harbor-database -d registry and pasted the password and it worked fine.
3. I ran the same previous command and tried an incorrect password and it failed as expected.

Current environment:

• k3s 1.22.5
• Harbor OSS version: v2.4.1
• Chart 1.8.1

Target environment:

• k3s 1.22.5 (unchanged)
• Harbor OSS version: v2.5.0
• Chart 1.9.0

Database: Internal, postgres

Error message:

2022-04-20T06:12:01Z [INFO] [/controller/artifact/annotation/parser.go:71]: the annotation parser to parser artifact annotation version v1alpha1 registered
2022-04-20T06:12:01Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.cncf.helm.config.v1+json registered
2022-04-20T06:12:01Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.cnab.manifest.v1 registered
2022-04-20T06:12:01Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.oci.image.index.v1+json registered
2022-04-20T06:12:01Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.docker.distribution.manifest.list.v2+json registered
2022-04-20T06:12:01Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.docker.distribution.manifest.v1+prettyjws registered
2022-04-20T06:12:01Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.oci.image.config.v1+json registered
2022-04-20T06:12:01Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.docker.container.image.v1+json registered
2022-04-20T06:12:01Z [INFO] [/pkg/reg/adapter/native/adapter.go:36]: the factory for adapter docker-registry registered
2022-04-20T06:12:01Z [INFO] [/pkg/reg/adapter/harbor/adaper.go:31]: the factory for adapter harbor registered
2022-04-20T06:12:01Z [INFO] [/pkg/reg/adapter/dockerhub/adapter.go:26]: Factory for adapter docker-hub registered
2022-04-20T06:12:01Z [INFO] [/pkg/reg/adapter/huawei/huawei_adapter.go:41]: the factory of Huawei adapter was registered
2022-04-20T06:12:01Z [INFO] [/pkg/reg/adapter/googlegcr/adapter.go:35]: the factory for adapter google-gcr registered
2022-04-20T06:12:01Z [INFO] [/pkg/reg/adapter/awsecr/adapter.go:43]: the factory for adapter aws-ecr registered
2022-04-20T06:12:01Z [INFO] [/pkg/reg/adapter/azurecr/adapter.go:15]: Factory for adapter azure-acr registered
2022-04-20T06:12:01Z [INFO] [/pkg/reg/adapter/aliacr/adapter.go:31]: the factory for adapter ali-acr registered
2022-04-20T06:12:01Z [INFO] [/pkg/reg/adapter/jfrog/adapter.go:46]: the factory of jfrog artifactory adapter was registered
2022-04-20T06:12:01Z [INFO] [/pkg/reg/adapter/quay/adapter.go:54]: the factory of Quay adapter was registered
2022-04-20T06:12:01Z [INFO] [/pkg/reg/adapter/helmhub/adapter.go:30]: the factory for adapter helm-hub registered
2022-04-20T06:12:01Z [INFO] [/pkg/reg/adapter/gitlab/adapter.go:17]: the factory for adapter gitlab registered
2022-04-20T06:12:01Z [INFO] [/pkg/reg/adapter/dtr/adapter.go:22]: the factory of dtr adapter was registered
2022-04-20T06:12:01Z [INFO] [/pkg/reg/adapter/artifacthub/adapter.go:30]: the factory for adapter artifact-hub registered
2022-04-20T06:12:01Z [INFO] [/pkg/reg/adapter/tencentcr/adapter.go:41]: the factory for adapter tencent-tcr registered
2022-04-20T06:12:01Z [INFO] [/pkg/reg/adapter/githubcr/adapter.go:29]: the factory for adapter github-ghcr registered
2022-04-20T06:12:01Z [INFO] [/core/controllers/base.go:155]: Config path: /etc/core/app.conf
2022-04-20T06:12:01Z [INFO] [/core/main.go:180]: initializing cache ...
2022-04-20T06:12:01Z [INFO] [/core/main.go:190]: initializing configurations...
2022-04-20T06:12:01Z [INFO] [/lib/config/systemconfig.go:197]: key path: /etc/core/key
2022-04-20T06:12:01Z [INFO] [/lib/config/config.go:92]: init secret store
2022-04-20T06:12:01Z [INFO] [/core/main.go:192]: configurations initialization completed
2022-04-20T06:12:01Z [INFO] [/common/dao/base.go:66]: Registering database: type-PostgreSQL host-harbor-database port-5432 database-registry sslmode-"disable"
[ORM]2022/04/20 06:12:01 register db Ping `default`, failed to connect to `host=harbor-database user=postgres database=registry`: server error (FATAL: password authentication failed for user "postgres" (SQLSTATE 28P01))
2022-04-20T06:12:01Z [FATAL] [/core/main.go:207]: failed to initialize database: register db Ping `default`, failed to connect to `host=harbor-database user=postgres database=registry`: server error (FATAL: password authentication failed for user "postgres" (SQLSTATE 28P01))

[zyyw]

please set enableMigrateHelmHook to false, and the db migration will be performed exactly as it was done in harbor 2.4.1

[zyyw]

According to the log message, it has to be invalid password. The odd thing is that the psql CLI can connect the db. @stonezdj could you take a look at it? thanks

[geneliu]

enableMigrateHelmHook to false does not change anything. Same failure: server error (FATAL: password authentication failed for user "postgres" (SQLSTATE 28P01)

[stonezdj]

@geneliu enableMigrateHelmHook will not run the pre-upgrade job, where is the server error (FATAL: password authentication failed for user "postgres" (SQLSTATE 28P01) found?

[geneliu]

@stonezdj , The error is from harbor-core pod (the pod keeps restarting from failure of authentication failed) after a helm upgrade (2.4.1 -> 2.5.0)

[geneliu]

@aureq wonder you have figured out any workaround?

[aureq]

When I had the failure, I decided not to upgrade to 2.5.0. In your case, if the upgrade job has failed, it may be safe for you to roll-back to your previous version while the Harbor team understand what's not working.

[geneliu]

Did the exact same. Just rolled back for now. Thanks!

[viceice]

@geneliu are you using the existing secret for external database? That is missing on the migration job env

https://github.com/goharbor/harbor-helm/issues/1214#issuecomment-1240502668

[thangamani-arun]

@viceice: For me, regardless of using password or existingSecret in vlaues.yaml file, the pre migration-jobs are still failing with redis user-name or password is wrong.

The failed upgrade was from 2.4.1 -> 2.6.3

[viceice]

@thangamani-arun that's why I'm using no auth for redis and using network policies to limit access to harbor only

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.

[viceice]

😕