PcapPlusPlus icon indicating copy to clipboard operation
PcapPlusPlus copied to clipboard

Add a benchmark which uses Google benchmark library

Open egecetin opened this issue 1 year ago • 7 comments

~~Automatically generate performance benchmark information for website using Google Benchmark~~

Add a google benchmark based benchmark application

egecetin avatar May 11 '24 19:05 egecetin

Codecov Report

Attention: Patch coverage is 0% with 108 lines in your changes missing coverage. Please review.

Project coverage is 83.15%. Comparing base (573effc) to head (e8efae5). Report is 1 commits behind head on dev.

Files with missing lines Patch % Lines
...amples/PcapPlusPlus-benchmark/benchmark-google.cpp 0.00% 108 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##              dev    #1392      +/-   ##
==========================================
- Coverage   83.35%   83.15%   -0.20%     
==========================================
  Files         276      277       +1     
  Lines       48086    48196     +110     
  Branches     9721     9697      -24     
==========================================
- Hits        40080    40077       -3     
- Misses       7115     7223     +108     
- Partials      891      896       +5     
Flag Coverage Δ
fedora40 75.19% <0.00%> (-0.22%) :arrow_down:
macos-13 80.65% <0.00%> (-0.21%) :arrow_down:
macos-14 80.65% <0.00%> (-0.21%) :arrow_down:
macos-15 80.62% <0.00%> (-0.22%) :arrow_down:
mingw32 70.87% <ø> (-0.05%) :arrow_down:
mingw64 70.88% <ø> (-0.01%) :arrow_down:
npcap 85.22% <ø> (-0.09%) :arrow_down:
rhel94 75.05% <0.00%> (-0.22%) :arrow_down:
ubuntu2004 58.60% <0.00%> (-0.15%) :arrow_down:
ubuntu2004-zstd 58.72% <0.00%> (-0.17%) :arrow_down:
ubuntu2204 74.96% <0.00%> (-0.21%) :arrow_down:
ubuntu2204-icpx 61.41% <ø> (+<0.01%) :arrow_up:
ubuntu2404 75.22% <0.00%> (-0.23%) :arrow_down:
unittest 83.15% <0.00%> (-0.20%) :arrow_down:
windows-2019 85.33% <ø> (-0.01%) :arrow_down:
windows-2022 85.35% <ø> (-0.02%) :arrow_down:
winpcap 85.32% <ø> (ø)
xdp 50.50% <0.00%> (-0.15%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Aug 30 '24 12:08 codecov[bot]

Where can I see the benchmark result?

tigercosmos avatar Aug 31 '24 12:08 tigercosmos

@tigercosmos Check "Run benchmark" step https://github.com/seladb/PcapPlusPlus/actions/runs/10644856460/job/29510043289

Approx. 10Gbps parsing, 1Gbps crafting performance in GitHub CI (just looking bytes per sec). Also items per sec is a good indicator which can we tabulate as for mean packet sizes (For example, if avg packet length 64, 512, 1024, 1500 etc -> size* items per sec *8

egecetin avatar Aug 31 '24 13:08 egecetin

Will it show a warning if the benchmark performance decreases too much? Otherwise, people will not click into the CI and check the result.

tigercosmos avatar Sep 01 '24 02:09 tigercosmos

@tigercosmos Actually my aim is to generate and update the benchmark results automatically. I enabled PR runs for just testing. I'll disable it when the work is done, and the runs will be only from master or dev. Because benchmark results can be very flaky without a dedicated machine/runner unfortunately. This workflow and integration with Google benchmark are just for easier benchmarking in local PCs and to give an idea about to performance in PcapPlusPlus website (since current benchmark really old)

egecetin avatar Sep 01 '24 06:09 egecetin

@egecetin I'm wondering if there's much value in publishing our benchmark without comparing PcapPlusPlus to other libraries. What do you think?

seladb avatar Oct 04 '24 06:10 seladb

@seladb Actually it is just for give an idea about the performance in terms of packet per second and for easier local benchmarking (both disk and cpu sides). Github's hardware is good enough to get good results. This is why i added long sentence to README and a workflow to CI. I agree it alone does not have true meaning unfortunately so why I left the old benchmark also. I can (at least might try 🙂) also update to comparing with other libraries periodically. Maybe once a month or at least manually run after every release in a different repo.

Maybe what we really need is to add this as another example (and move the instructions to the internal README)? It's probably not worth running it in CI and uploading the results to the web-site repo

seladb avatar Oct 09 '24 08:10 seladb

@seladb It is ready to review if you have time

egecetin avatar Nov 11 '24 07:11 egecetin