harbor-helm icon indicating copy to clipboard operation
harbor-helm copied to clipboard

Published 20 hours ago • verified publisher icon goharbor

Can not scan image with proxy setting

Open hillbun opened this issue 2 years ago • 5 comments

I am behind firewall, I set http_proxy and https_proxy to deployment harbor-jobservice

      - env:
        - name: http_proxy
          value: http://x.x.x.x:3128
        - name: https_proxy
          value: http://x.x.x.x:3128

I can access https://ghcr.io/v2/ when enter pod of harbor-jobservice with curl

curl https://ghcr.io/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required"}]}

But when I scan images with harbor UI, it still return network problem.

2022-05-07T03:30:15Z [ERROR] [/pkg/scan/job.go:292]: check scan report with mime type application/vnd.security.vulnerability.report; version=1.1: running trivy wrapper: running trivy: exit status 1: 2022-05-07T03:29:43.947Z	[34mINFO[0m	Need to update DB
2022-05-07T03:29:43.947Z	[34mINFO[0m	Downloading DB...
2022-05-07T03:30:13.169Z	[31mFATAL[0m	DB error: failed to download vulnerability DB: OCI artifact error: OCI artifact error: OCI repository error: Get "https://ghcr.io/v2/": dial tcp 20.205.243.164:443: i/o timeout
: general response handler: unexpected status code: 500, expected: 200

hillbun avatar May 07 '22 03:05 hillbun

I setup proxy:

proxy: httpProxy: http://x.x.x.x:3128 httpsProxy: http://x.x.x.x:3128 noProxy: 127.0.0.1,localhost,.local,.internal,x.x.0.0/16 components: - core - jobservice - trivy

But still get errors:

2022-05-10T03:46:34Z [INFO] [/pkg/scan/job.go:385]: {
  "uuid": "e925753e-ac0c-11ec-94eb-f6987478469d",
  "name": "Trivy",
  "description": "The Trivy scanner adapter",
  "url": "http://harbor-trivy:8080",
  "disabled": false,
  "is_default": true,
  "health": "healthy",
  "auth": "",
  "access_credential": "[HIDDEN]",
  "skip_certVerify": false,
  "use_internal_addr": true,
  "adapter": "Trivy",
  "vendor": "Aqua Security",
  "version": "v0.24.2",
  "create_time": "2022-03-25T07:26:39.277575Z",
  "update_time": "2022-03-25T07:26:39.277577Z"
}
2022-05-10T03:46:34Z [INFO] [/pkg/scan/job.go:385]: {
  "registry": {
    "url": "http://harbor-core:80",
    "authorization": "[HIDDEN]"
  },
  "artifact": {
    "namespace_id": 9,
    "repository": "tbox/tct2tbox-h5",
    "tag": "9bb70b2fb52dbe79065315b5c880f9ba08569272",
    "digest": "sha256:f9f311e67e780d98ec862ff451ba36a5affd16d6edc6254782e931aeaa60f1c6",
    "mime_type": "application/vnd.docker.distribution.manifest.v2+json"
  }
}
2022-05-10T03:46:34Z [INFO] [/pkg/scan/job.go:167]: Report mime types: [application/vnd.security.vulnerability.report; version=1.1]
2022-05-10T03:46:34Z [INFO] [/pkg/scan/job.go:222]: Get report for mime type: application/vnd.security.vulnerability.report; version=1.1
2022-05-10T03:46:36Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:46:41Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:46:46Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:46:51Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:46:56Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:47:01Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:47:06Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:47:11Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:47:16Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:47:21Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:47:26Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:47:31Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:47:36Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:47:41Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:47:46Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:47:51Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:47:56Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:48:01Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:48:06Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:48:11Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:48:16Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:48:21Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:48:26Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:48:31Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:48:36Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:48:41Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:48:46Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:48:51Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:48:56Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:49:01Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:49:06Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:49:11Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:49:16Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:49:21Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:49:26Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:49:31Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:49:36Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:49:41Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:49:46Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:49:51Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:49:56Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:50:01Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:50:06Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:50:11Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:50:16Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:50:21Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:50:26Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:50:31Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:50:36Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:50:41Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:50:46Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:50:51Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:50:56Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:51:01Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:51:06Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:51:11Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:51:16Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:51:21Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:51:26Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:51:31Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:51:36Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2022-05-10T03:51:41Z [ERROR] [/pkg/scan/job.go:292]: check scan report with mime type application/vnd.security.vulnerability.report; version=1.1: running trivy wrapper: running trivy: exit status 1: 2022-05-10T03:51:38.975Z	[31mFATAL[0m	scan error: image scan failed: failed analysis: analyze error: timeout: context deadline exceeded
: general response handler: unexpected status code: 500, expected: 200

hillbun avatar May 10 '22 04:05 hillbun 

{"exit_code":1,"level":"error","msg":"Running trivy failed","std_out":"Incorrect Usage. flag provided but not defined: -format\n\nNAME:\n   trivy - A simple and comprehensive vulnerability scanner for containers\n\nUSAGE:\n   trivy [global options] command [command options] target\n\nVERSION:\n   0.24.2\n\nCOMMANDS:\n   image, i          scan an image\n   filesystem, fs    scan local filesystem for language-specific dependencies and config files\n   rootfs            scan rootfs\n   repository, repo  scan remote repository\n   client, c         client mode\n   server, s         server mode\n   config, conf      scan config files\n   plugin, p         manage plugins\n   help, h           Shows a list of commands or help for one command\n\nGLOBAL OPTIONS:\n   --quiet, -q        suppress progress bar and log output (default: false) [$TRIVY_QUIET]\n   --debug, -d        debug mode (default: false) [$TRIVY_DEBUG]\n   --cache-dir value  cache directory (default: \"/home/scanner/.cache/trivy\") [$TRIVY_CACHE_DIR]\n   --help, -h         show help (default: false)\n   --version, -v      print the version (default: false)\n2022-05-10T05:46:53.172Z\t\u001b[31mFATAL\u001b[0m\tflag provided but not defined: -format\n","time":"2022-05-10T05:46:53Z"}
{"error":"running trivy: exit status 1: Incorrect Usage. flag provided but not defined: -format\n\nNAME:\n   trivy - A simple and comprehensive vulnerability scanner for containers\n\nUSAGE:\n   trivy [global options] command [command options] target\n\nVERSION:\n   0.24.2\n\nCOMMANDS:\n   image, i          scan an image\n   filesystem, fs    scan local filesystem for language-specific dependencies and config files\n   rootfs            scan rootfs\n   repository, repo  scan remote repository\n   client, c         client mode\n   server, s         server mode\n   config, conf      scan config files\n   plugin, p         manage plugins\n   help, h           Shows a list of commands or help for one command\n\nGLOBAL OPTIONS:\n   --quiet, -q        suppress progress bar and log output (default: false) [$TRIVY_QUIET]\n   --debug, -d        debug mode (default: false) [$TRIVY_DEBUG]\n   --cache-dir value  cache directory (default: \"/home/scanner/.cache/trivy\") [$TRIVY_CACHE_DIR]\n   --help, -h         show help (default: false)\n   --version, -v      print the version (default: false)\n2022-05-10T05:46:53.172Z\t\u001b[31mFATAL\u001b[0m\tflag provided but not defined: -format\n","level":"error","msg":"Error while retrieving vulnerability DB version","time":"2022-05-10T05:46:53Z"}
{"exit_code":-1,"image_ref":"harbor-core:80/tbox/tct-admin@sha256:41627ee14894ed045adb31197c00bd5f6f2319cf7bc8b3b995b7ecee4a6b7e49","level":"error","msg":"Running trivy failed","std_out":"","time":"2022-05-10T05:47:14Z"}
{"error":"running trivy wrapper: running trivy: signal: killed: ","level":"error","msg":"Scan failed","time":"2022-05-10T05:47:14Z"}
{"error":"running trivy wrapper: running trivy: signal: killed: ","level":"error","msg":"Scan job failed","scan_job_id":"1d0d29c3a0648123c30e64d0","time":"2022-05-10T05:47:16Z"}

hillbun avatar May 10 '22 05:05 hillbun

Hi @hillbun, for this below error:

"error":"running trivy: exit status 1: Incorrect Usage. flag provided but not defined: -format\n\nNAME:\n

please ignore it. This is a known issue of trivy, and this error should have no impact on scan. More details of this error:

  • https://github.com/goharbor/harbor/issues/16554

The other errors may due to your config of proxy.

zyyw avatar May 19 '22 03:05 zyyw

@hillbun could you try to config proxy to

proxy:
httpProxy: http://x.x.x.x:3128/
httpsProxy: http://x.x.x.x:3128/
noProxy: 127.0.0.1,localhost,.local,.internal,x.x.0.0/16
components:
- trivy

instead of:

proxy:
httpProxy: http://x.x.x.x:3128/
httpsProxy: http://x.x.x.x:3128/
noProxy: 127.0.0.1,localhost,.local,.internal,x.x.0.0/16
components:
- core
- jobservice
- trivy

zyyw avatar May 19 '22 07:05 zyyw

I confirm this works. We also need a proxy and have deployed harbor with the above mentioned proxy parameters, instead of env. variables.

hasanhakkaev avatar Jun 21 '22 07:06 hasanhakkaev

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

github-actions[bot] avatar Feb 08 '24 09:02 github-actions[bot]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.

github-actions[bot] avatar Mar 11 '24 09:03 github-actions[bot]