retention error because of pq to many clients.

[Jeremy-boo]

Expected behavior and actual behavior:

If the tag retention rule is set under the Harbor project to run every day, when the number of artifacts under the Harbor project reaches a certain scale, harbor-core will fill up the number of connections to the postgrtes database, resulting in temporary unavailability of harbor

project numbers: 1000+

Number of artifacts under the project: 10000+

harbor error info:

current harbor database max_connetion: 1024

database error info:

Versions:

• harbor version: [v2.2.3]
• docker engine version: [20.10.7]
• k8s version: [v1.23.13]

Additional context: after review harbor retaion source code, I think that the database connection will be regenerated when the event events are generated concurrently. If the number of events reaches a certain number, the number of database connections will be filled.

https://github.com/goharbor/harbor/blob/74ecf354cdc15f667368b69426a3d138cd40e8e4/src/pkg/notifier/notifier.go#L190

• Harbor config files: You can get them by packaging harbor.yml and files in the same directory, including subdirectory.
• Log files: You can get them by package the /var/log/harbor/ .

2022-11-10T06:08:24Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: driver: bad connection"}]}
2022-11-10T06:08:24Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: driver: bad connection"}]}
2022-11-10T06:08:24Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: driver: bad connection"}]}
2022-11-10T06:08:24Z [ERROR] [/lib/orm/orm.go:78]: rollback transaction failed: driver: bad connection
2022-11-10T06:08:24Z [ERROR] [/lib/orm/orm.go:78]: rollback transaction failed: driver: bad connection
2022-11-10T06:08:24Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: deal with /service/notifications/tasks/7434 request in transaction failed: driver: bad connection"}]}
2022-11-10T06:08:24Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: deal with /service/notifications/tasks/7549 request in transaction failed: driver: bad connection"}]}
2022-11-10T06:08:24Z [ERROR] [/lib/orm/orm.go:78]: rollback transaction failed: driver: bad connection
2022-11-10T06:08:24Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: deal with /service/notifications/tasks/7523 request in transaction failed: driver: bad connection"}]}
2022-11-10T06:08:24Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: driver: bad connection"}]}
2022-11-10T06:08:24Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: driver: bad connection"}]}
2022-11-10T06:08:24Z [ERROR] [/pkg/retention/callback.go:49][error="driver: bad connection" requestID="ff05ade5-6710-4758-92d0-47c8b78f5abf"]: failed to update of retention task 7462
2022-11-10T06:08:24Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: driver: bad connection"}]}
2022-11-10T06:08:24Z [ERROR] [/pkg/retention/callback.go:49][error="driver: bad connection" requestID="11abbee1-c0a3-4095-a9e4-4674602a6595"]: failed to update of retention task 7309
2022-11-10T06:08:24Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: driver: bad connection"}]}
2022-11-10T06:08:24Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: driver: bad connection"}]}
2022-11-10T06:08:24Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: driver: bad connection"}]}
2022-11-10T06:08:24Z [ERROR] [/lib/orm/orm.go:78]: rollback transaction failed: driver: bad connection
2022-11-10T06:08:24Z [ERROR] [/lib/orm/orm.go:78]: rollback transaction failed: driver: bad connection
2022-11-10T06:08:24Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: deal with /service/notifications/tasks/7352 request in transaction failed: driver: bad connection"}]}
2022-11-10T06:08:24Z [ERROR] [/pkg/retention/callback.go:49][error="driver: bad connection" requestID="d7a03538-a218-4f28-9eca-4516385730df"]: failed to update of retention task 7529

[MinerYang]

Have you tried to set the db max_connections larger than 1024 https://github.com/goharbor/harbor/wiki/Harbor-FAQs#modify-the-max-connections-of-the-database Harbor v2.2.3 is legacy , we have keep improving performance related problems and you could upgrade Harbor and try on the latest version

[Jeremy-boo]

Have you tried to set the db max_connections larger than 1024 https://github.com/goharbor/harbor/wiki/Harbor-FAQs#modify-the-max-connections-of-the-database Harbor v2.2.3 is legacy , we have keep improving performance related problems and you could upgrade Harbor and try on the latest version

yeah, increasing the number of database connections can temporarily solve the problem, but if there are orders of magnitude more artifacts, there will still be a problem because too many connections will affect other functions of Harbor. Although this problem is not fatal, it is indeed a point that harbor retains tag rules and is worth optimizing

[chlins]

@Jeremy-boo Sure, we did some enhancements for harbor tag retention recently, and you could try to upgrade harbor to v2.6+ or waiting for the v2.7.

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[danielfbm]

@Jeremy-boo Sure, we did some enhancements for harbor tag retention recently, and you could try to upgrade harbor to v2.6+ or waiting for the v2.7.

@chlins can you please confirm the fixed version so we can close this issue? thanks

[chlins]

@Jeremy-boo Sure, we did some enhancements for harbor tag retention recently, and you could try to upgrade harbor to v2.6+ or waiting for the v2.7.

@chlins can you please confirm the fixed version so we can close this issue? thanks

@danielfbm The latest version v2.7.0 should have resolved this issue.

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.