collector
collector copied to clipboard
Re-enable benchmark CI step
Description
Currently benchmark CI step is disabled. Turn it on with following changes:
- Use berserker instread of hackbench
- Use regular ttest instead of IQR
- Compare CPU/Memory usage instead of hackbench time
#run-benchmark
Checklist
- [ ] Investigated and inspected CI test results
- [ ] Updated documentation accordingly
Automated testing
- [ ] Added unit tests
- [ ] Added integration tests
- [ ] Added regression tests
If any of these don't apply, please comment below.
Testing Performed
Manual testing.
| VM | Method | Baseline CPU median (%) | Test CPU median (%) | CPU P-value |
|---|---|---|---|---|
| rhel.rhel-8 | core-bpf | 32.38 | 34.44 | :green_circle: |
| rhel.rhel-8 | ebpf | 24.56 | 24.95 | :green_circle: |
| rhel.rhel-9 | core-bpf | 42.4 | 42.63 | :green_circle: |
| rhel.rhel-9 | ebpf | 22.88 | 66.97 | :red_circle: |
| ubuntu-os.ubuntu-2004-lts | core-bpf | 45.59 | 44.06 | :green_circle: |
| ubuntu-os.ubuntu-2004-lts | ebpf | 32.02 | 35.28 | :green_circle: |
| ubuntu-os.ubuntu-2204-lts | core-bpf | 46.82 | 43.67 | :green_circle: |
| ubuntu-os.ubuntu-2204-lts | ebpf | 34.22 | 33.38 | :green_circle: |
| VM | Method | Baseline Memory median (MiB) | Test Memory median (MiB) | Memory P-value |
|---|---|---|---|---|
| rhel.rhel-8 | core-bpf | 48.55 | 47.1 | :green_circle: |
| rhel.rhel-8 | ebpf | 30.52 | 34.68 | :green_circle: |
| rhel.rhel-9 | core-bpf | 51.94 | 50.92 | :green_circle: |
| rhel.rhel-9 | ebpf | 36.18 | 94.13 | :red_circle: |
| ubuntu-os.ubuntu-2004-lts | core-bpf | 51.07 | 49.66 | :green_circle: |
| ubuntu-os.ubuntu-2004-lts | ebpf | 29.23 | 29.51 | :green_circle: |
| ubuntu-os.ubuntu-2204-lts | core-bpf | 53.07 | 52.54 | :green_circle: |
| ubuntu-os.ubuntu-2204-lts | ebpf | 30.52 | 29.44 | :green_circle: |
Just a heads up on the socat integration test failures - the socat image is using alpine:latest which pulls in a new version of socat that seems to consistently fail (I've not looked into precisely what's wrong with it) but we can pin the image to alpine:3.18.4 to stick with the old and working version.
I had to do this in the syscall latency branch as well: https://github.com/stackrox/collector/pull/1640/commits/65cf13545a4bcdb1225448672b9c9ae195286c6f