harbor
harbor copied to clipboard
goharbor
Many restart harbor-jobservice or CrashLoopbackoff on v2.5.3
Hello,
We just installed a fresh version of Harbor v2.5.3 but we encounter an issue with the pod "harbor-jobservice"
harbor-jobservice
2022-08-10T03:56:21Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"IMAGE_SCAN","id":"659802af222d106b05baf954","t":1660090012,"args":null}
2022-08-10T03:56:21Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-08-10T03:56:21Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-08-10T03:56:24Z [ERROR] [/jobservice/runner/redis.go:122]: Job 'IMAGE_SCAN:bf8b8dcc7855228d69912fa3' exit with error: run error: check scan report with mime type application/vnd.security.vulnerability.report; version=1.1: cannot find scan job: 28898633903e9f90a95a1b48: general response handler: unexpected status code: 404, expected: 200
2022-08-10T03:56:24Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"IMAGE_SCAN","id":"5b87b511a04eb9baa59444c7","t":1660090013,"args":null}
2022-08-10T03:56:24Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-08-10T03:56:24Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-08-10T03:56:25Z [ERROR] [/jobservice/runner/redis.go:122]: Job 'IMAGE_SCAN:5e57e676367a45d5619c0036' exit with error: run error: check scan report with mime type application/vnd.security.vulnerability.report; version=1.1: cannot find scan job: f84ace7bb3333c23edf795fe: general response handler: unexpected status code: 404, expected: 200
2022-08-10T03:56:25Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"IMAGE_SCAN","id":"d2208bd4b543d8a5619e95a4","t":1660090013,"args":null}
2022-08-10T03:56:25Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-08-10T03:56:25Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-08-10T03:56:26Z [ERROR] [/jobservice/runner/redis.go:122]: Job 'IMAGE_SCAN:5b87b511a04eb9baa59444c7' exit with error: run error: check scan report with mime type application/vnd.security.vulnerability.report; version=1.1: cannot find scan job: d5cd3d6f4e3758ff38767fe7: general response handler: unexpected status code: 404, expected: 200
2022-08-10T03:56:26Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"IMAGE_SCAN","id":"519cf7477b2264fb35953f88","t":1660090014,"args":null}
2022-08-10T03:56:26Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-08-10T03:56:26Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-08-10T03:56:27Z [ERROR] [/jobservice/runner/redis.go:122]: Job 'IMAGE_SCAN:d2208bd4b543d8a5619e95a4' exit with error: run error: check scan report with mime type application/vnd.security.vulnerability.report; version=1.1: cannot find scan job: 57c5f7dd897435a1ae7b52d6: general response handler: unexpected status code: 404, expected: 200
2022-08-10T08:26:35Z [INFO] [/jobservice/sync/schedule.go:248]: Found 10 schedules, restore 0 missing schedules, clear 0 dirty schedules
2022-08-10T08:26:35Z [ERROR] [/jobservice/runner/redis.go:93]: Track job 1da9c8d14f9d5026fd64d326: stats may not have been ready yet, hold for 1055 ms and retry again
2022-08-10T08:26:36Z [ERROR] [/jobservice/runner/redis.go:93]: Track job 71e0e2185d91fd6903d5d691: stats may not have been ready yet, hold for 1055 ms and retry again
2022-08-10T08:26:36Z [INFO] [/jobservice/logger/sweeper_controller.go:97]: 0 outdated log entries are sweepped by sweeper *sweeper.FileSweeper
2022-08-10T08:26:37Z [ERROR] [/jobservice/runner/redis.go:93]: Track job 1da9c8d14f9d5026fd64d326: stats may not have been ready yet, hold for 1388 ms and retry again
2022-08-10T08:26:37Z [ERROR] [/jobservice/runner/redis.go:93]: Track job 71e0e2185d91fd6903d5d691: stats may not have been ready yet, hold for 1388 ms and retry again
2022-08-10T08:26:38Z [ERROR] [/jobservice/runner/redis.go:93]: Track job 1da9c8d14f9d5026fd64d326: stats may not have been ready yet, hold for 1499 ms and retry again
2022-08-10T08:26:38Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-08-10T08:26:38Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-08-10T08:26:39Z [ERROR] [/jobservice/runner/redis.go:93]: Track job 1da9c8d14f9d5026fd64d326: stats may not have been ready yet, hold for 1388 ms and retry again
2022-08-10T08:26:40Z [INFO] [/jobservice/worker/cworker/reaper.go:88]: Start: Reap in-progress jobs from the dead pools
2022-08-10T08:26:40Z [INFO] [/jobservice/worker/cworker/reaper.go:126]: End: Reap in-progress jobs
2022-08-10T08:26:41Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
2022-08-10T08:26:41Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://harbor-core:80/api/v2.0/internalconfig
panic: interface conversion: interface {} is nil, not []uint8
goroutine 410 [running]:
github.com/goharbor/harbor/src/jobservice/job.getStatus({0x2fbcc70, 0xc000c18f18}, {0xc00143e730, 0x18})
/harbor/src/jobservice/job/tracker.go:631 +0x194
github.com/goharbor/harbor/src/jobservice/job.(*basicTracker).Status(0xc0008c0320)
/harbor/src/jobservice/job/tracker.go:193 +0xc7
github.com/goharbor/harbor/src/jobservice/job/impl.(*Context).OPCommand(0x1a3185c5000)
/harbor/src/jobservice/job/impl/context.go:160 +0x26
github.com/goharbor/harbor/src/pkg/scan.(*Job).Run.func1()
/harbor/src/pkg/scan/job.go:151 +0x38
github.com/goharbor/harbor/src/pkg/scan.(*Job).Run.func2(0x0, {0xc00076c180, 0x3a})
/harbor/src/pkg/scan/job.go:231 +0x347
created by github.com/goharbor/harbor/src/pkg/scan.(*Job).Run
/harbor/src/pkg/scan/job.go:218 +0x132f
Can anyone explain me what is it ?
THANKS
@BK-STAR77 this seems like not a harbor-helm issue. Please open a issue in this repo:
- https://github.com/goharbor/harbor
FYI, this issue also has similar error message stats may not have been ready yet:
- https://github.com/goharbor/harbor/issues/17347
@chlins could you please take a look on this issue. We may at least to fix the panic scenario.
For people who have a problem , we have on our side changed all the pods to replicas=1 and that solved our problem.