Failed to push artifacts during load

[kesslerdev]

Expected behavior and actual behavior:

When a load occurs (triggered by several push pipelines from GitLab) on Harbor, several pushes are successful in a decent amount of time (<10s), while others can hang for 10 to 30 minutes and fail. However, all pulls work.

We have scaled the core and registry from 3 to 6, with no significant improvements. Additionally, we don't have a very high load (maximum 40 operations per second for all API calls) with an average of 5 operations per second.

When the problem occurs, we can have 10 calls of PATCH /v2/project/image/blobs/uploads terminated with errors in over 600 seconds.

Versions: Please specify the versions of following systems.

• harbor version: [2.7.2]
• kubernetes version: [1.26.13]
• istio version: [1.18.3]

Additional context:

• When Harbor encounters issues with pushes:
  • The database is hardly processing any requests.
  • The S3 storage (backed by physical bay) used by Harbor and several applications (such as GitLab) does not show any problematic load, as "all indicators are green."
  • The built-in Redis is also idle.

Graphes from grafana: From Harbor: For v2_blob_upload PATCH & v2_blob_upload PUT

Also v2_blob_upload PATCH & v2_blob_upload PUT but for all core nodes

From istio:

Namespace pods:

NAME                                  READY   STATUS    RESTARTS       AGE
harbor-chartmuseum-7b479dd6d6-56m5g   1/1     Running   0              6d22h
harbor-core-6f7b48c949-btmw6          1/1     Running   1 (5d6h ago)   5d6h
harbor-core-6f7b48c949-k8zh8          1/1     Running   1 (5d6h ago)   5d6h
harbor-core-6f7b48c949-kx2gp          1/1     Running   1 (5d6h ago)   5d6h
harbor-core-6f7b48c949-lhtqc          1/1     Running   1 (5d6h ago)   5d6h
harbor-core-6f7b48c949-sww2m          1/1     Running   1 (5d6h ago)   5d6h
harbor-core-6f7b48c949-wzjc8          1/1     Running   1 (5d6h ago)   5d6h
harbor-exporter-5fc8975d8b-fhmjx      1/1     Running   0              21d
harbor-jobservice-544996775-cjnp9     1/1     Running   0              5d5h
harbor-nginx-df7bf48bc-xgw4n          1/1     Running   0              2d23h
harbor-nginx-df7bf48bc-zl9zs          1/1     Running   0              2d23h
harbor-portal-77786765fc-gvgt2        1/1     Running   0              6d22h
harbor-portal-77786765fc-j6kh2        1/1     Running   0              6d22h
harbor-portal-77786765fc-t7b97        1/1     Running   0              6d22h
harbor-redis-0                        1/1     Running   0              5d6h
harbor-registry-86579b6bcd-5jkbw      2/2     Running   0              5d6h
harbor-registry-86579b6bcd-b25pz      2/2     Running   0              5d6h
harbor-registry-86579b6bcd-f6wcg      2/2     Running   0              5d6h
harbor-registry-86579b6bcd-gbrbj      2/2     Running   0              5d6h
harbor-registry-86579b6bcd-kg2zt      2/2     Running   0              5d6h
harbor-registry-86579b6bcd-mh7wq      2/2     Running   0              5d6h
harbor-trivy-0                        1/1     Running   0              4d6h

• Harbor config files:
  • Installed by helm harbor chart harbor-1.11.2

caBundleSecretName: harbor-crt
caSecretName: harbor-crt
cache:
  enabled: false
chartmuseum:
  enabled: true
core:
  configureUserSettings: '{"auth_mode":"oidc_auth","http_authproxy_admin_groups":"","http_authproxy_endpoint":"","http_authproxy_server_certificate":"","http_authproxy_skip_search":false,"http_authproxy_tokenreview_endpoint":"","http_authproxy_verify_cert":true,"notification_enable":true,"oidc_admin_group":"ADM_GRP","oidc_auto_onboard":true,"oidc_client_id":"CLIENTID","oidc_endpoint":"OIDC","oidc_extra_redirect_parms":{},"oidc_groups_claim":"GROUPS","oidc_name":"NAME","oidc_scope":"SCOPES","oidc_user_claim":"CLAIM","oidc_verify_cert":true,"project_creation_restriction":"adminonly","quota_per_project_enable":true,"read_only":false,"robot_name_prefix":"robot-","robot_token_duration":356,"self_registration":false,"storage_per_project":53687091200,"token_expiration":30,"oidc_client_secret":"SECRET"}'
  replicas: 3
  secretName: harbor-crt
database:
  external:
    coreDatabase: registry
    host: PG_HOST
    notaryServerDatabase: notary_server
    notarySignerDatabase: notary_signer
    password: PG_PASSWORD
    port: "PG_PORT"
    sslmode: disable
    username: PG_USERNAME
  maxIdleConns: 100
  maxOpenConns: 900
  type: external
enableMigrateHelmHook: false
expose:
  ingress:
    hosts:
      core: hub.***.***.com
  tls:
    certSource: secret
    enabled: true
    secret:
      secretName: harbor-crt-tls
  type: ClusterIP
externalURL: https://hub.***.***.com
harborAdminPassword: ADMIN_PASSWORD
internalTLS:
  enabled: false
logLevel: info
metrics:
  enabled: true
  serviceMonitor:
    additionalLabels:
      release: prom-app
    enabled: true
notary:
  enabled: false
persistence:
  enabled: true
  imageChartStorage:
    caBundleSecretName: harbor-crt
    disableredirect: true
    s3:
      accesskey: S3_ACCESSKEY
      bucket: S3_BUCKET
      encrypt: false
      region: us-east-1
      regionendpoint: S3_ENDPOINT
      rootdirectory: /
      secretkey: S3_SECRET
      secure: true
      v4auth: true
    type: s3
  persistentVolumeClaim:
    chartmuseum:
      accessMode: ReadWriteOnce
      size: 5Gi
    database:
      accessMode: ReadWriteOnce
      size: 1Gi
    jobservice:
      jobLog:
        accessMode: ReadWriteOnce
        size: 10Gi
        storageClass: rook
    redis:
      accessMode: ReadWriteOnce
      size: 5Gi
      storageClass: rook
    registry:
      accessMode: ReadWriteOnce
      size: 5Gi
    trivy:
      accessMode: ReadWriteOnce
      size: 5Gi
      storageClass: rook
  resourcePolicy: keep
portal:
  replicas: 3
proxy:
  components:
  - core
  - jobservice
  - trivy
  httpProxy: <PROXY>
  httpsProxy: <PROXY>
  noProxy: <NO_PROXY>
redis:
  type: internal
registry:
  replicas: 3
secretKey: <SECRET_KEY>
trace:
  enabled: false
trivy:
  enabled: true

logs.zip

[chlins]

Could you please attach the harbor core and registry logs as well?

[kesslerdev]

Sure i've added the logs of core & registry pods in the issue

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.