harbor icon indicating copy to clipboard operation
harbor copied to clipboard

Published 20 hours ago • verified publisher icon goharbor

harbor-database: terminating connection due to unexpected postmaster exit

Open tangjiaxing669 opened this issue 2 years ago • 2 comments

I use helm to install harbor, I used helm to install harbor, and the next day I found that pods kept restarting. I don't know why, I saw on the Internet that modifying registry.relativeurls to true can fix the problem (I don't know what this option means), I tried it, but the problem is still the same

harbor-chartmuseum-5fbb6ff764-b9rd6                       1/1     Running   0          24h
harbor-core-6df8b9665c-wvb8k                              1/1     Running   2          24h
harbor-database-0                                         1/1     Running   31         7h41m
harbor-jobservice-5554fbfcb4-65lrc                        1/1     Running   3          24h
harbor-nginx-784db44d9d-cz4sv                             1/1     Running   0          2d3h
harbor-notary-server-7fb96d49dd-pgqqs                     1/1     Running   33         24h
harbor-notary-signer-777bc47f74-9spg4                     1/1     Running   0          24h
harbor-portal-bc45ddf47-nddbr                             1/1     Running   0          2d3h
harbor-redis-0                                            1/1     Running   0          2d3h
harbor-registry-547d776758-xzmps                          2/2     Running   0          24h
harbor-trivy-0                                            1/1     Running   0          2d3h

/ # kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:04:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}

# harbor 1.10.0 [2.6.0]

The values.yaml:

expose:
  type: nodePort
  tls:
    enabled: true
    certSource: secret
    auto:
      commonName: ''
    secret:
      secretName: harbor-cert-secret
      notarySecretName: ''
  ingress:
    hosts:
      core: core.harbor.domain
      notary: notary.harbor.domain
    controller: default
    kubeVersionOverride: ''
    className: ''
    annotations:
      ingress.kubernetes.io/ssl-redirect: 'true'
      ingress.kubernetes.io/proxy-body-size: '0'
      nginx.ingress.kubernetes.io/ssl-redirect: 'true'
      nginx.ingress.kubernetes.io/proxy-body-size: '0'
    notary:
      annotations: {}
      labels: {}
    harbor:
      annotations: {}
      labels: {}
  clusterIP:
    name: harbor
    annotations: {}
    ports:
      httpPort: 80
      httpsPort: 443
      notaryPort: 4443
  nodePort:
    name: harbor
    ports:
      http:
        port: 80
        nodePort: 30002
      https:
        port: 443
        nodePort: 30003
      notary:
        port: 4443
        nodePort: 30004
  loadBalancer:
    name: harbor
    IP: ''
    ports:
      httpPort: 80
      httpsPort: 443
      notaryPort: 4443
    annotations: {}
    sourceRanges: []
externalURL: 'https://harbor.phzt.weextest.com'
internalTLS:
  enabled: false
  certSource: auto
  trustCa: ''
  core:
    secretName: ''
    crt: ''
    key: ''
  jobservice:
    secretName: ''
    crt: ''
    key: ''
  registry:
    secretName: ''
    crt: ''
    key: ''
  portal:
    secretName: ''
    crt: ''
    key: ''
  chartmuseum:
    secretName: ''
    crt: ''
    key: ''
  trivy:
    secretName: ''
    crt: ''
    key: ''
ipFamily:
  ipv6:
    enabled: true
  ipv4:
    enabled: true
persistence:
  enabled: true
  resourcePolicy: keep
  persistentVolumeClaim:
    registry:
      existingClaim: ''
      storageClass: local
      subPath: ''
      accessMode: ReadWriteOnce
      size: 500Gi
      annotations: {}
    chartmuseum:
      existingClaim: ''
      storageClass: local
      subPath: ''
      accessMode: ReadWriteOnce
      size: 5Gi
      annotations: {}
    jobservice:
      jobLog:
        existingClaim: ''
        storageClass: local
        subPath: ''
        accessMode: ReadWriteOnce
        size: 1Gi
        annotations: {}
      scanDataExports:
        existingClaim: ''
        storageClass: local
        subPath: ''
        accessMode: ReadWriteOnce
        size: 1Gi
        annotations: {}
    database:
      existingClaim: ''
      storageClass: local
      subPath: ''
      accessMode: ReadWriteOnce
      size: 1Gi
      annotations: {}
    redis:
      existingClaim: ''
      storageClass: local
      subPath: ''
      accessMode: ReadWriteOnce
      size: 1Gi
      annotations: {}
    trivy:
      existingClaim: ''
      storageClass: local
      subPath: ''
      accessMode: ReadWriteOnce
      size: 5Gi
      annotations: {}
  imageChartStorage:
    disableredirect: false
    type: filesystem
    filesystem:
      rootdirectory: /storage
    azure:
      accountname: accountname
      accountkey: base64encodedaccountkey
      container: containername
      existingSecret: ''
    gcs:
      bucket: bucketname
      encodedkey: base64-encoded-json-key-file
      existingSecret: ''
      useWorkloadIdentity: false
    s3:
      region: us-west-1
      bucket: bucketname
    swift:
      authurl: 'https://storage.myprovider.com/v3/auth'
      username: username
      password: password
      container: containername
    oss:
      accesskeyid: accesskeyid
      accesskeysecret: accesskeysecret
      region: regionname
      bucket: bucketname
imagePullPolicy: IfNotPresent
imagePullSecrets: null
updateStrategy:
  type: RollingUpdate
logLevel: info
harborAdminPassword: Harbor12345
caSecretName: ''
secretKey: not-a-secure-key
existingSecretSecretKey: ''
proxy:
  httpProxy: null
  httpsProxy: null
  noProxy: '127.0.0.1,localhost,.local,.internal'
  components:
    - core
    - jobservice
    - trivy
enableMigrateHelmHook: false
nginx:
  image:
    repository: goharbor/nginx-photon
    tag: v2.6.0
  serviceAccountName: ''
  automountServiceAccountToken: false
  replicas: 1
  revisionHistoryLimit: 10
  nodeSelector: {}
  tolerations: []
  affinity: {}
  podAnnotations: {}
  priorityClassName: null
portal:
  image:
    repository: goharbor/harbor-portal
    tag: v2.6.0
  serviceAccountName: ''
  automountServiceAccountToken: false
  replicas: 1
  revisionHistoryLimit: 10
  nodeSelector: {}
  tolerations: []
  affinity: {}
  podAnnotations: {}
  priorityClassName: null
core:
  image:
    repository: goharbor/harbor-core
    tag: v2.6.0
  serviceAccountName: ''
  automountServiceAccountToken: false
  replicas: 1
  revisionHistoryLimit: 10
  startupProbe:
    enabled: true
    initialDelaySeconds: 10
  nodeSelector: {}
  tolerations: []
  affinity: {}
  podAnnotations: {}
  secret: ''
  secretName: ''
  xsrfKey: ''
  priorityClassName: null
  artifactPullAsyncFlushDuration: null
  gdpr:
    deleteUser: false
jobservice:
  image:
    repository: goharbor/harbor-jobservice
    tag: v2.6.0
  replicas: 1
  revisionHistoryLimit: 10
  serviceAccountName: ''
  automountServiceAccountToken: false
  maxJobWorkers: 10
  jobLoggers:
    - file
  loggerSweeperDuration: 14
  nodeSelector: {}
  tolerations: []
  affinity: {}
  podAnnotations: {}
  secret: ''
  priorityClassName: null
registry:
  serviceAccountName: ''
  automountServiceAccountToken: false
  registry:
    image:
      repository: goharbor/registry-photon
      tag: v2.6.0
  controller:
    image:
      repository: goharbor/harbor-registryctl
      tag: v2.6.0
  replicas: 1
  revisionHistoryLimit: 10
  nodeSelector: {}
  tolerations: []
  affinity: {}
  podAnnotations: {}
  priorityClassName: null
  secret: ''
  relativeurls: true
  credentials:
    username: harbor_registry_user
    password: harbor_registry_password
    existingSecret: ''
  middleware:
    enabled: false
    type: cloudFront
    cloudFront:
      baseurl: example.cloudfront.net
      keypairid: KEYPAIRID
      duration: 3000s
      ipfilteredby: none
      privateKeySecret: my-secret
  upload_purging:
    enabled: true
    age: 168h
    interval: 24h
    dryrun: false
chartmuseum:
  enabled: true
  serviceAccountName: ''
  automountServiceAccountToken: false
  absoluteUrl: false
  image:
    repository: goharbor/chartmuseum-photon
    tag: v2.6.0
  replicas: 1
  revisionHistoryLimit: 10
  nodeSelector: {}
  tolerations: []
  affinity: {}
  podAnnotations: {}
  priorityClassName: null
  indexLimit: 0
trivy:
  enabled: true
  image:
    repository: goharbor/trivy-adapter-photon
    tag: v2.6.0
  serviceAccountName: ''
  automountServiceAccountToken: false
  replicas: 1
  debugMode: false
  vulnType: 'os,library'
  severity: 'UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL'
  ignoreUnfixed: false
  insecure: false
  gitHubToken: ''
  skipUpdate: false
  offlineScan: false
  timeout: 5m0s
  resources:
    requests:
      cpu: 200m
      memory: 512Mi
    limits:
      cpu: 1
      memory: 1Gi
  nodeSelector: {}
  tolerations: []
  affinity: {}
  podAnnotations: {}
  priorityClassName: null
notary:
  enabled: true
  server:
    serviceAccountName: ''
    automountServiceAccountToken: false
    image:
      repository: goharbor/notary-server-photon
      tag: v2.6.0
    replicas: 1
    nodeSelector: {}
    tolerations: []
    affinity: {}
    podAnnotations: {}
    priorityClassName: null
  signer:
    serviceAccountName: ''
    automountServiceAccountToken: false
    image:
      repository: goharbor/notary-signer-photon
      tag: v2.6.0
    replicas: 1
    nodeSelector: {}
    tolerations: []
    affinity: {}
    podAnnotations: {}
    priorityClassName: null
  secretName: ''
database:
  type: internal
  internal:
    serviceAccountName: ''
    automountServiceAccountToken: false
    image:
      repository: goharbor/harbor-db
      tag: v2.6.0
    password: changeit
    shmSizeLimit: 512Mi
    nodeSelector: {}
    tolerations: []
    affinity: {}
    priorityClassName: null
    initContainer:
      migrator: {}
      permissions: {}
  external:
    host: 192.168.0.1
    port: '5432'
    username: user
    password: password
    coreDatabase: registry
    notaryServerDatabase: notary_server
    notarySignerDatabase: notary_signer
    existingSecret: ''
    sslmode: disable
  maxIdleConns: 100
  maxOpenConns: 900
  podAnnotations: {}
redis:
  type: internal
  internal:
    serviceAccountName: ''
    automountServiceAccountToken: false
    image:
      repository: goharbor/redis-photon
      tag: v2.6.0
    nodeSelector: {}
    tolerations: []
    affinity: {}
    priorityClassName: null
  external:
    addr: '192.168.0.2:6379'
    sentinelMasterSet: ''
    coreDatabaseIndex: '0'
    jobserviceDatabaseIndex: '1'
    registryDatabaseIndex: '2'
    chartmuseumDatabaseIndex: '3'
    trivyAdapterIndex: '5'
    password: ''
    existingSecret: ''
  podAnnotations: {}
exporter:
  replicas: 1
  revisionHistoryLimit: 10
  podAnnotations: {}
  serviceAccountName: ''
  automountServiceAccountToken: false
  image:
    repository: goharbor/harbor-exporter
    tag: v2.6.0
  nodeSelector: {}
  tolerations: []
  affinity: {}
  cacheDuration: 23
  cacheCleanInterval: 14400
  priorityClassName: null
metrics:
  enabled: false
  core:
    path: /metrics
    port: 8001
  registry:
    path: /metrics
    port: 8001
  jobservice:
    path: /metrics
    port: 8001
  exporter:
    path: /metrics
    port: 8001
  serviceMonitor:
    enabled: false
    additionalLabels: {}
    interval: ''
    metricRelabelings: []
    relabelings: []
trace:
  enabled: false
  provider: jaeger
  sample_rate: 1
  jaeger:
    endpoint: 'http://hostname:14268/api/traces'
  otel:
    endpoint: 'hostname:4318'
    url_path: /v1/traces
    compression: false
    insecure: true
    timeout: 10s
cache:
  enabled: false
  expireHours: 24

Below is the harbor-database-0 log information from loki:

2022-09-16 04:29:58.139 UTC [596] FATAL:  the database system is shutting down
2022-09-16 04:29:47.802 UTC [585] FATAL:  the database system is shutting down
2022-09-16 04:29:37.832 UTC [573] FATAL:  the database system is shutting down
2022-09-16 04:29:27.460 UTC [562] FATAL:  the database system is shutting down
2022-09-16 04:29:17.891 UTC [552] FATAL:  the database system is shutting down
2022-09-16 04:29:08.459 UTC [540] FATAL:  the database system is shutting down
2022-09-16 04:28:57.605 UTC [529] FATAL:  the database system is shutting down
2022-09-16 04:28:47.649 UTC [518] FATAL:  the database system is shutting down
2022-09-16 04:28:37.847 UTC [507] FATAL:  the database system is shutting down
2022-09-16 04:28:27.595 UTC [496] FATAL:  the database system is shutting down
2022-09-16 04:28:17.394 UTC [486] FATAL:  the database system is shutting down
2022-09-16 04:28:08.568 UTC [475] FATAL:  the database system is shutting down
2022-09-16 04:27:58.191 UTC [1] LOG:  received smart shutdown request
2022-09-16 04:22:14.429 UTC [1] LOG:  database system is ready to accept connections
2022-09-16 04:22:14.392 UTC [8] LOG:  redo done at 0/21B9228
2022-09-16 04:22:14.392 UTC [8] LOG:  invalid record length at 0/21B9260: wanted 24, got 0
2022-09-16 04:22:14.392 UTC [8] LOG:  redo starts at 0/21B9228
2022-09-16 04:22:14.386 UTC [8] LOG:  database system was not properly shut down; automatic recovery in progress
2022-09-16 04:22:14.037 UTC [8] LOG:  database system was interrupted; last known up at 2022-09-16 04:14:18 UTC
2022-09-16 04:22:13.968 UTC [1] LOG:  listening on Unix socket "/run/postgresql/.s.PGSQL.5432"
2022-09-16 04:22:13.894 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2022-09-16 04:22:13.894 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2022-09-16 04:22:13.893 UTC [1] LOG:  starting PostgreSQL 13.8 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 10.2.0, 64-bit
no need to upgrade postgres, launch it.
2022-09-16 04:40:17.693 UTC [914] FATAL:  the database system is shutting down
2022-09-16 04:40:08.693 UTC [902] FATAL:  the database system is shutting down
2022-09-16 04:39:57.647 UTC [892] FATAL:  the database system is shutting down
2022-09-16 04:39:47.446 UTC [880] FATAL:  the database system is shutting down
2022-09-16 04:39:37.961 UTC [869] FATAL:  the database system is shutting down
2022-09-16 04:39:27.633 UTC [859] FATAL:  the database system is shutting down
2022-09-16 04:39:17.994 UTC [848] FATAL:  the database system is shutting down
2022-09-16 04:39:08.548 UTC [837] FATAL:  the database system is shutting down
2022-09-16 04:38:57.490 UTC [827] FATAL:  the database system is shutting down
2022-09-16 04:38:47.508 UTC [813] FATAL:  the database system is shutting down
2022-09-16 04:38:37.561 UTC [802] FATAL:  the database system is shutting down
2022-09-16 04:38:27.538 UTC [792] FATAL:  the database system is shutting down
2022-09-16 04:38:18.285 UTC [1] LOG:  received smart shutdown request
2022-09-16 04:30:05.304 UTC [1] LOG:  database system is ready to accept connections
2022-09-16 04:30:04.125 UTC [9] LOG:  redo done at 0/21B92D8
2022-09-16 04:30:04.125 UTC [9] LOG:  invalid record length at 0/21B9310: wanted 24, got 0
2022-09-16 04:30:04.125 UTC [9] LOG:  redo starts at 0/21B92D8
2022-09-16 04:30:04.075 UTC [9] LOG:  database system was not properly shut down; automatic recovery in progress
2022-09-16 04:30:02.853 UTC [9] LOG:  database system was interrupted; last known up at 2022-09-16 04:22:14 UTC
2022-09-16 04:30:02.743 UTC [1] LOG:  listening on Unix socket "/run/postgresql/.s.PGSQL.5432"
2022-09-16 04:30:02.527 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2022-09-16 04:30:02.527 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2022-09-16 04:30:02.513 UTC [1] LOG:  starting PostgreSQL 13.8 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 10.2.0, 64-bit
no need to upgrade postgres, launch it.
2022-09-16 04:51:58.614 UTC [37] FATAL:  terminating connection due to unexpected postmaster exit
2022-09-16 04:51:58.614 UTC [81] FATAL:  terminating connection due to unexpected postmaster exit
2022-09-16 04:51:57.786 UTC [1078] FATAL:  the database system is shutting down
2022-09-16 04:51:47.639 UTC [1068] FATAL:  the database system is shutting down
2022-09-16 04:51:37.361 UTC [1057] FATAL:  the database system is shutting down
2022-09-16 04:51:27.508 UTC [1046] FATAL:  the database system is shutting down
2022-09-16 04:51:18.134 UTC [1034] FATAL:  the database system is shutting down
2022-09-16 04:51:08.333 UTC [1022] FATAL:  the database system is shutting down
2022-09-16 04:50:57.321 UTC [1011] FATAL:  the database system is shutting down
2022-09-16 04:50:47.509 UTC [999] FATAL:  the database system is shutting down
2022-09-16 04:50:37.528 UTC [988] FATAL:  the database system is shutting down
2022-09-16 04:50:27.433 UTC [978] FATAL:  the database system is shutting down
2022-09-16 04:50:17.795 UTC [966] FATAL:  the database system is shutting down
2022-09-16 04:50:07.879 UTC [955] FATAL:  the database system is shutting down
2022-09-16 04:49:58.587 UTC [945] FATAL:  the database system is shutting down
2022-09-16T12:49:58+08:00	2022-09-16 04:49:58.384 UTC [1] LOG:  received smart shutdown request
2022-09-16T12:40:23+08:00	2022-09-16 04:40:23.379 UTC [1] LOG:  database system is ready to accept connections
2022-09-16T12:40:22+08:00	2022-09-16 04:40:22.633 UTC [8] LOG:  redo done at 0/21B9388
2022-09-16T12:40:22+08:00	2022-09-16 04:40:22.633 UTC [8] LOG:  invalid record length at 0/21B93C0: wanted 24, got 0
2022-09-16T12:40:22+08:00	2022-09-16 04:40:22.633 UTC [8] LOG:  redo starts at 0/21B9388
2022-09-16T12:40:22+08:00	2022-09-16 04:40:22.612 UTC [8] LOG:  database system was not properly shut down; automatic recovery in progress
2022-09-16T12:40:21+08:00	2022-09-16 04:40:21.510 UTC [8] LOG:  database system was interrupted; last known up at 2022-09-16 04:30:05 UTC
2022-09-16T12:40:21+08:00	2022-09-16 04:40:21.468 UTC [1] LOG:  listening on Unix socket "/run/postgresql/.s.PGSQL.5432"
2022-09-16T12:40:21+08:00	2022-09-16 04:40:21.448 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2022-09-16T12:40:21+08:00	2022-09-16 04:40:21.445 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2022-09-16T12:40:21+08:00	2022-09-16 04:40:21.444 UTC [1] LOG:  starting PostgreSQL 13.8 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 10.2.0, 64-bit
2022-09-16T12:40:21+08:00	no need to upgrade postgres, launch it.
2022-09-16T13:00:19+08:00	2022-09-16 05:00:19.016 UTC [35] FATAL:  terminating connection due to unexpected postmaster exit
2022-09-16T13:00:19+08:00	2022-09-16 05:00:19.009 UTC [37] FATAL:  terminating connection due to unexpected postmaster exit

And harbor-notary-server has the same problem, Below is log info:

 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"83b62206-5d40-4bde-bb61-4316fbd87999","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:38196","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"24.253043ms","http.response.status":404,"http.response.written":116,"level":"info","msg":"response completed","time":"2022-09-16T03:30:13Z"}
 {"go.version":"go1.14.15","harbor.phzt.weextest.com/weex-dev/boss-web":"gun","http.request.host":"harbor-notary-server:4443","http.request.id":"6cdfa74b-03a8-4f68-9738-3562fafb2b08","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:38242","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"404 GET root role","time":"2022-09-16T03:30:13Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"6cdfa74b-03a8-4f68-9738-3562fafb2b08","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:38242","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"metadata not found: You have requested metadata that does not exist.: No record found","time":"2022-09-16T03:30:13Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"6cdfa74b-03a8-4f68-9738-3562fafb2b08","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:38242","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"13.209529ms","http.response.status":404,"http.response.written":116,"level":"info","msg":"response completed","time":"2022-09-16T03:30:13Z"}
 {"go.version":"go1.14.15","harbor.phzt.weextest.com/weex-dev/boss-web":"gun","http.request.host":"harbor-notary-server:4443","http.request.id":"1eeadda9-5bc2-4442-b23c-4cd129aae748","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50810","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"404 GET root role","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"1eeadda9-5bc2-4442-b23c-4cd129aae748","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50810","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"metadata not found: You have requested metadata that does not exist.: No record found","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"1eeadda9-5bc2-4442-b23c-4cd129aae748","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50810","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"12.200694ms","http.response.status":404,"http.response.written":116,"level":"info","msg":"response completed","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","harbor.phzt.weextest.com/weex-dev/boss-web":"gun","http.request.host":"harbor-notary-server:4443","http.request.id":"d1c5123f-ce4e-4410-b0f5-70e48fdc8a37","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50814","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"404 GET root role","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"d1c5123f-ce4e-4410-b0f5-70e48fdc8a37","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50814","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"metadata not found: You have requested metadata that does not exist.: No record found","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"d1c5123f-ce4e-4410-b0f5-70e48fdc8a37","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50814","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"18.053922ms","http.response.status":404,"http.response.written":116,"level":"info","msg":"response completed","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","harbor.phzt.weextest.com/weex-dev/boss-web":"gun","http.request.host":"harbor-notary-server:4443","http.request.id":"5582e002-b808-47f0-990d-39ebe4686834","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50812","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"404 GET root role","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"5582e002-b808-47f0-990d-39ebe4686834","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50812","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"metadata not found: You have requested metadata that does not exist.: No record found","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"5582e002-b808-47f0-990d-39ebe4686834","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50812","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"36.142037ms","http.response.status":404,"http.response.written":116,"level":"info","msg":"response completed","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","harbor.phzt.weextest.com/weex-dev/boss-web":"gun","http.request.host":"harbor-notary-server:4443","http.request.id":"3f210bbc-cc31-4134-b562-9866a5b7297d","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50816","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"404 GET root role","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"3f210bbc-cc31-4134-b562-9866a5b7297d","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50816","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"metadata not found: You have requested metadata that does not exist.: No record found","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"3f210bbc-cc31-4134-b562-9866a5b7297d","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50816","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"28.284131ms","http.response.status":404,"http.response.written":116,"level":"info","msg":"response completed","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","harbor.phzt.weextest.com/weex-dev/boss-web":"gun","http.request.host":"harbor-notary-server:4443","http.request.id":"f9a2f3b0-f235-4bc7-9487-b58259ce8c40","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50818","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"404 GET root role","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"f9a2f3b0-f235-4bc7-9487-b58259ce8c40","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50818","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"metadata not found: You have requested metadata that does not exist.: No record found","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"f9a2f3b0-f235-4bc7-9487-b58259ce8c40","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50818","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"10.352333ms","http.response.status":404,"http.response.written":116,"level":"info","msg":"response completed","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","harbor.phzt.weextest.com/weex-dev/boss-web":"gun","http.request.host":"harbor-notary-server:4443","http.request.id":"9d33216b-a1a8-4150-971a-47a93e59850e","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50820","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"404 GET root role","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"9d33216b-a1a8-4150-971a-47a93e59850e","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50820","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"metadata not found: You have requested metadata that does not exist.: No record found","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"9d33216b-a1a8-4150-971a-47a93e59850e","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50820","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"6.776602ms","http.response.status":404,"http.response.written":116,"level":"info","msg":"response completed","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","harbor.phzt.weextest.com/weex-dev/boss-web":"gun","http.request.host":"harbor-notary-server:4443","http.request.id":"aa6e20ff-0faa-49c2-85af-549896ed34ee","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50824","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"404 GET root role","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"aa6e20ff-0faa-49c2-85af-549896ed34ee","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50824","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"metadata not found: You have requested metadata that does not exist.: No record found","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"aa6e20ff-0faa-49c2-85af-549896ed34ee","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50824","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"4.521239ms","http.response.status":404,"http.response.written":116,"level":"info","msg":"response completed","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","harbor.phzt.weextest.com/weex-dev/boss-web":"gun","http.request.host":"harbor-notary-server:4443","http.request.id":"f5d80a1e-6436-4f51-9e38-441edff84994","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50826","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"404 GET root role","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"f5d80a1e-6436-4f51-9e38-441edff84994","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50826","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"metadata not found: You have requested metadata that does not exist.: No record found","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"f5d80a1e-6436-4f51-9e38-441edff84994","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50826","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"4.906011ms","http.response.status":404,"http.response.written":116,"level":"info","msg":"response completed","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","harbor.phzt.weextest.com/weex-dev/boss-web":"gun","http.request.host":"harbor-notary-server:4443","http.request.id":"3b54acdc-92ff-4b9b-a6f4-57ddea3ed250","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50828","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"404 GET root role","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"3b54acdc-92ff-4b9b-a6f4-57ddea3ed250","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50828","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"metadata not found: You have requested metadata that does not exist.: No record found","time":"2022-09-16T03:39:22Z"}
 {"go.version":"go1.14.15","http.request.host":"harbor-notary-server:4443","http.request.id":"3b54acdc-92ff-4b9b-a6f4-57ddea3ed250","http.request.method":"GET","http.request.remoteaddr":"10.233.95.179:50828","http.request.uri":"/v2/harbor.phzt.weextest.com/weex-dev/boss-web/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"7.327546ms","http.response.status":404,"http.response.written":116,"level":"info","msg":"response completed","time":"2022-09-16T03:39:22Z"}

tangjiaxing669 avatar Sep 16 '22 10:09 tangjiaxing669

It seems your database is in the abnormal status, the db logs shows it tries to recover but failed, the database maybe have the dirty data, so if your harbor is a fresh and no data in it, you could try to clean PVC and re-install it.

chlins avatar Sep 19 '22 01:09 chlins

@chlins Thank you for your answer. I tried it, I removed all pvc and reinstalled harbor, but the problem is still the same.

tangjiaxing669 avatar Sep 19 '22 02:09 tangjiaxing669

It seems your database restart every 20 minutes, do you have any cron jobs do some operations around the terminate time?

chlins avatar Sep 26 '22 08:09 chlins

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

github-actions[bot] avatar Nov 25 '22 09:11 github-actions[bot]

I have the exact same issue with the exact same pods.

The behaviour I'm experiencing is that the pods become extremely slow to respond - I believe Kube is killing the pods because they don't respond to the liveness/readiness checks within 1 second.

The weirdest thing is, the pods don't really seem to be doing anything. Almost no CPU/memory usage in metrics yet when I shell into the pod, the database pod in particular, it is very lumpy and slow in performance.

Deploying onto EKS in my case, Kube 1.21.

rossbishop avatar Dec 07 '22 11:12 rossbishop

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

github-actions[bot] avatar Feb 06 '23 09:02 github-actions[bot]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.

github-actions[bot] avatar Mar 09 '23 09:03 github-actions[bot]