aquasecurity
Operator blocked by jobs with error: unexpected end of JSON input
What steps did you take and what happened:
The trivy operator does not remove some jobs from vulnerability reports and maybe stops working because of the scanJobsConcurrentLimit.
The problem seems to occur in jobs with prometheus-metrics-collector only.
the pod has the status completed but inside of it the prometheus-metrics-collector container has status Error but no logs, unfortunately.
The job has the condition BackoffLimitExceeded: True.
What did you expect to happen:
Trivy should succeed to scan this pod or should at least not get blocked by it.
Anything else you would like to add:
The error message (every few minutes):
2025-11-10T11:13:43Z ERROR reconciler.scan job Scan job container {"job": "trivy-system/scan-vulnerabilityreport-7bcf774cc9", "container": "prometheus-metrics-collector", "status.reason": "Error", "status.message": ""}
github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).completedContainers
/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:441
github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).SetupWithManager.(*ScanJobController).reconcileJobs.func1
/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:103
sigs.k8s.io/controller-runtime/pkg/reconcile.TypedFunc[...].Reconcile
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:134
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:461
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:421
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:296
2025-11-10T11:13:43Z DEBUG reconciler.scan job Job complete {"job": "trivy-system/scan-vulnerabilityreport-7bcf774cc9", "kind": "DaemonSet", "name": "gke-metrics-agent-scaling-500", "namespace": "kube-system", "podSpecHash": "79b8669b89"}
2025-11-10T11:13:43Z ERROR Reconciler error {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "Job": {"name":"scan-vulnerabilityreport-7bcf774cc9","namespace":"trivy-system"}, "namespace": "trivy-system", "name": "scan-vulnerabilityreport-7bcf774cc9", "reconcileID": "1cc80bd2-8978-4d1f-a8e5-1b514f4f1fa8", "error": "unexpected end of JSON input"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:474
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:421
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:296
We deployed the trivy operator via the helm chart with ArgoCD. Our values.yaml (we use trivy as a subchart):
trivy-operator:
operator:
scanJobsConcurrentLimit: 2
infraAssessmentScannerEnabled: false
controllerCacheSyncTimeout: 30m
logDevMode: true
podAnnotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '8080'
trivy:
ignoreUnfixed: true
resources:
requests:
memory: 750M
limits:
memory: 750M
Environment:
- Trivy-Operator version: 0.29.0
- Kubernetes version: v1.33.5-gke.1125000
- OS: Container-Optimized OS from Google 6.6.97+ on n1-standard-2
@alexander-dammeier thanks for the report!
could you pls try to set scanJob.compressLogs to false?
