collector icon indicating copy to clipboard operation
collector copied to clipboard

Re-enable benchmark CI step

Open erthalion opened this issue 1 year ago • 2 comments

Description

Currently benchmark CI step is disabled. Turn it on with following changes:

  • Use berserker instread of hackbench
  • Use regular ttest instead of IQR
  • Compare CPU/Memory usage instead of hackbench time

#run-benchmark

Checklist

  • [ ] Investigated and inspected CI test results
  • [ ] Updated documentation accordingly

Automated testing

  • [ ] Added unit tests
  • [ ] Added integration tests
  • [ ] Added regression tests

If any of these don't apply, please comment below.

Testing Performed

Manual testing.

erthalion avatar May 03 '24 11:05 erthalion

VM Method Baseline CPU median (%) Test CPU median (%) CPU P-value
rhel.rhel-8 core-bpf 32.38 34.44 :green_circle:
rhel.rhel-8 ebpf 24.56 24.95 :green_circle:
rhel.rhel-9 core-bpf 42.4 42.63 :green_circle:
rhel.rhel-9 ebpf 22.88 66.97 :red_circle:
ubuntu-os.ubuntu-2004-lts core-bpf 45.59 44.06 :green_circle:
ubuntu-os.ubuntu-2004-lts ebpf 32.02 35.28 :green_circle:
ubuntu-os.ubuntu-2204-lts core-bpf 46.82 43.67 :green_circle:
ubuntu-os.ubuntu-2204-lts ebpf 34.22 33.38 :green_circle:


VM Method Baseline Memory median (MiB) Test Memory median (MiB) Memory P-value
rhel.rhel-8 core-bpf 48.55 47.1 :green_circle:
rhel.rhel-8 ebpf 30.52 34.68 :green_circle:
rhel.rhel-9 core-bpf 51.94 50.92 :green_circle:
rhel.rhel-9 ebpf 36.18 94.13 :red_circle:
ubuntu-os.ubuntu-2004-lts core-bpf 51.07 49.66 :green_circle:
ubuntu-os.ubuntu-2004-lts ebpf 29.23 29.51 :green_circle:
ubuntu-os.ubuntu-2204-lts core-bpf 53.07 52.54 :green_circle:
ubuntu-os.ubuntu-2204-lts ebpf 30.52 29.44 :green_circle:

github-actions[bot] avatar May 08 '24 16:05 github-actions[bot]

Just a heads up on the socat integration test failures - the socat image is using alpine:latest which pulls in a new version of socat that seems to consistently fail (I've not looked into precisely what's wrong with it) but we can pin the image to alpine:3.18.4 to stick with the old and working version.

I had to do this in the syscall latency branch as well: https://github.com/stackrox/collector/pull/1640/commits/65cf13545a4bcdb1225448672b9c9ae195286c6f

Stringy avatar May 09 '24 09:05 Stringy