fleet
fleet copied to clipboard
fleetdm
Vulnerability Processing Taking Longer than expected
Fleet version: 4.44.0
Web browser and operating system: NA
💥 Actual behavior
Vulnerability processing was observed taking 2x longer after upgrade to v4.44.0 in some Fleet Cloud instances.
🧑💻 Steps to reproduce
Slack thread reference: https://fleetdm.slack.com/archives/C019WG4GH0A/p1707682626450799
This was observed with CPU utilization and spotchecked observing created_at and updated_at timestamps for vulnerability cron jobs in the database.
🕯️ More info (optional)
An increase in scan time is expected when upgrading to 4.44.0 due to the addition of OS vulnerability scans, but 2x seems excessive. As observed in one cloud hosted environment, the total OS count was ~40, and primarily macOS devices.
QA Testplan
- Run vuln processing before this change and note the vulns identified.
- Run vuln processing after this change and compare to the above.
Logs:
CloudWatch Logs Insights
region: us-east-2
log-group-names: REDACTED
start-time: -21600s
end-time: 0s
query-string:
fields @timestamp, @message, @logStream, @log
| sort @timestamp desc
| filter cron like /vulnerabilities/
| filter msg not like /might still be running/
| filter msg not like /error translating to CPE/
| limit 5
| @timestamp | @message | @logStream | @log |
|---|---|---|---|
| 2024-02-14 21:39:58.087 | {"cron":"vulnerabilities","instanceID":"kGeGzhC5+fL4M6y1vEhkW05A4wTYi8Fhw0az3hXLpnT9G//NcPRIsO+3yzIqGGJPjVCH17ebvsT5Z8ncSeRQ+A==","level":"info","msg":"total runtime (2h39m41.05537752s) exceeded schedule interval (1h0m0s)","schedule":"vulnerabilities","ts":"2024-02-14T21:39:58.086985161Z"} | fleet/fleet/11addbc9bdc84b3483ed075a8f2a5f17 | 611884880216:pinterest |
| 2024-02-14 21:39:58.086 | {"cron":"vulnerabilities","instanceID":"kGeGzhC5+fL4M6y1vEhkW05A4wTYi8Fhw0az3hXLpnT9G//NcPRIsO+3yzIqGGJPjVCH17ebvsT5Z8ncSeRQ+A==","level":"info","schedule":"vulnerabilities","status":"completed","ts":"2024-02-14T21:39:58.086847177Z"} | fleet/fleet/11addbc9bdc84b3483ed075a8f2a5f17 | 611884880216:pinterest |
| 2024-02-14 19:00:17.038 | {"cron":"vulnerabilities","instanceID":"kGeGzhC5+fL4M6y1vEhkW05A4wTYi8Fhw0az3hXLpnT9G//NcPRIsO+3yzIqGGJPjVCH17ebvsT5Z8ncSeRQ+A==","level":"info","schedule":"vulnerabilities","status":"pending","ts":"2024-02-14T19:00:17.037640489Z"} | fleet/fleet/11addbc9bdc84b3483ed075a8f2a5f17 | 611884880216:pinterest |
| 2024-02-14 19:00:17.038 | {"cron":"vulnerabilities","level":"info","periodicity":"1h0m0s","ts":"2024-02-14T19:00:17.037686982Z"} | fleet/fleet/11addbc9bdc84b3483ed075a8f2a5f17 | 611884880216:pinterest |
| 2024-02-14 19:00:17.038 | {"cron":"vulnerabilities","level":"info","msg":"scanning vulnerabilities","ts":"2024-02-14T19:00:17.037712783Z"} | fleet/fleet/11addbc9bdc84b3483ed075a8f2a5f17 | 611884880216:pinterest |
@mostlikelee Do CVEs and CPEs always have a vendor (they never have a full/partial wildcard vendor)? What about the product?
I couldn't think of a haiku this time. (See fleetdm.com logs for more information.)
