New scanner version (v0.30.0) consumes too much CPU

[hasonhai]

What steps did you take and what happened:

We install the new scanner version v0.30.0 and do a full scan in our Harbor with around 8200 images. The old version (v0.27.0) took 2.5 hours to finish. CPU used is little. I have 3 instances of it and each only consume 0.25 core in average. The new version (v0.30.0) took 6 hours and just finished 1200 images. CPU used is quite high. I have 3 instances of its and each consume full 2 cores. Both of the version were deployed with the same settings and works with the same Harbor registry.

CPU usage of v0.27.0

CPU usage of v0.30.0

Memory usage of them are similar and low. Network throughput is also not stand out.

What did you expect to happen:

I expected that the CPU consumption difference does not have this large gap.

Environment:

• Harbor version: v2.4.1
• Harbor Scanner Adapter for Trivy version: v0.30.0
• Harbor installation process: harbor-helm

[hasonhai]

I would like to add more info: Increasing CPU limit to 5 not help much. the first 6 hours with 2 cores, 1000 images scanned later in 10 hours with 5 cores, 3800 images scanned more

Here is our config:

      SCANNER_LOG_LEVEL:                      info
      SCANNER_API_SERVER_ADDR:                :8080
      SCANNER_API_SERVER_READ_TIMEOUT:        15s
      SCANNER_API_SERVER_WRITE_TIMEOUT:       15s
      SCANNER_API_SERVER_IDLE_TIMEOUT:        60s
      SCANNER_TRIVY_CACHE_DIR:                /home/scanner/.cache/trivy
      SCANNER_TRIVY_REPORTS_DIR:              /home/scanner/.cache/reports
      SCANNER_TRIVY_DEBUG_MODE:               false
      SCANNER_TRIVY_VULN_TYPE:                os,library
      SCANNER_TRIVY_SEVERITY:                 UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
      SCANNER_TRIVY_IGNORE_UNFIXED:           false
      SCANNER_TRIVY_TIMEOUT:                  20m0s
      SCANNER_TRIVY_SKIP_UPDATE:              true
      SCANNER_TRIVY_OFFLINE_SCAN:             true
      SCANNER_TRIVY_GITHUB_TOKEN:             <set to the key 'gitHubToken' in secret 'trivy-v0-30-0-harbor-scanner-trivy'>  Optional: false
      SCANNER_TRIVY_INSECURE:                 true
      SCANNER_STORE_REDIS_NAMESPACE:          harbor.scanner.trivy.v0.30.0:store
      SCANNER_STORE_REDIS_SCAN_JOB_TTL:       1h
      SCANNER_JOB_QUEUE_REDIS_NAMESPACE:      harbor.scanner.trivy.v0.30.0:job-queue
      SCANNER_JOB_QUEUE_WORKER_CONCURRENCY:   10
      SCANNER_REDIS_URL:                      redis://abcxyz-harbor-redis:6379
      SCANNER_REDIS_POOL_MAX_ACTIVE:          10
      SCANNER_REDIS_POOL_MAX_IDLE:            10
      SCANNER_REDIS_POOL_IDLE_TIMEOUT:        5m
      SCANNER_REDIS_POOL_CONNECTION_TIMEOUT:  1s
      SCANNER_REDIS_POOL_READ_TIMEOUT:        1s
      SCANNER_REDIS_POOL_WRITE_TIMEOUT:       1s

We use offlineScan and have a cronjob to update the CVE database for us. Sometimes we have images with size more than 10GBs and take long time to complete so we change the Trivy timeout to 20m.

[hasonhai]

We did the same deploy of the scanner on a different cluster. This cluster is bigger but the Harbor Registry that runs on it has only ~1200 images. The CPU consumption issue is the same.

[lixdavid94]

@hasonhai I noticed the same behavior with the trivy scans taking longer after updating to v0.30.0. I think it's because of secret scanning which was introduced in trivy v0.27.0. I've since rolled back to adapter version v0.28.0 which uses trivy v0.26.0 that doesn't have the secret scanning feature.

I have this issue opened to see if there is a way to disable secret scanning.