FirebirdSQL
Adds .NET distributed tracing instrumentation & metrics.
Included the final portion of #1196 (metrics) here, as it requires the same infrastructure as traces.
Here is before and after comparison of Perf benchmark results on my machine.
BenchmarkDotNet=v0.13.2, OS=Windows 11 (10.0.26120.2130)
AMD Ryzen 9 5950X, 1 CPU, 32 logical and 16 physical cores
.NET SDK=9.0.100-rc.2.24474.11
[Host] : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2
NuGet : .NET 6.0.35 (6.0.3524.45918), X64 RyuJIT AVX2
Project : .NET 6.0.35 (6.0.3524.45918), X64 RyuJIT AVX2
Jit=RyuJit Platform=X64 Toolchain=.NET 6.0
WarmupCount=3
Perf.CommandBenchmark
FirebirdSQL:master (Old) <-> fdcastel:add-telemetry-trace (New)
| Diff | Method | Mean | Error | Allocated |
|---|---|---|---|---|
| Old | Execute | 40,075.4 μs | 1,450.87 μs | 313.71 KB |
| New | 34,322.0 μs (-14%) | 658.70 μs | 313.11 KB (0%) | |
| Old | Execute | 34,465.9 μs | 680.65 μs | 311.3 KB |
| New | 34,248.1 μs (-1%) | 384.64 μs | 315.91 KB (+1%) | |
| Old | Fetch | 640.1 μs | 11.02 μs | 56.88 KB |
| New | 648.4 μs (+1%) | 12.19 μs | 56.88 KB (0%) | |
| Old | Fetch | 653.2 μs | 10.90 μs | 51.43 KB |
| New | 653.1 μs (0%) | 9.41 μs | 56.06 KB (+9%) | |
| Old | Execute | 33,690.3 μs | 648.64 μs | 319.86 KB |
| New | 33,559.9 μs (0%) | 664.54 μs | 319.85 KB (0%) | |
| Old | Execute | 32,943.5 μs | 235.67 μs | 317.28 KB |
| New | 34,278.5 μs (+4%) | 683.34 μs | 322.01 KB (+1%) | |
| Old | Fetch | 642.2 μs | 3.47 μs | 60.75 KB |
| New | 647.3 μs (+1%) | 12.89 μs | 60.75 KB (0%) | |
| Old | Fetch | 664.2 μs | 7.49 μs | 55.31 KB |
| New | 658.7 μs (-1%) | 8.82 μs | 59.92 KB (+8%) |
Thanks @willibrandon. This surely can be improved.
Could you kindly share the steps/configuration you used for the above benchmark results?
Mine here produces a comparison between Release (built from local sources) and ReleaseNuget (NuGet package for FirebirdClient 8.5.4).
@willibrandon I pushed a new PR in #1203 (it is based on this PR).
Could you please clone it and run ./run-benchmark.ps1 in your system?
The script will compare the current project (built from sources with .NET8) with the latest NuGet release (10.3.1).
BenchmarkDotNet v0.14.0, Windows 11 (10.0.22621.4317/22H2/2022Update/SunValley2)
13th Gen Intel Core i7-13700T, 1 CPU, 24 logical and 16 physical cores
.NET SDK 9.0.100-rc.2.24474.11
[Host] : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2
Core80 : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2
NuGet80 : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2
Jit=RyuJit Platform=X64 Toolchain=.NET 8.0
WarmupCount=3
| Method | Job | BuildConfiguration | DataType | Count | Mean | Error | StdDev | Ratio | RatioSD | Gen0 | Allocated | Alloc Ratio |
|-------- |-------- |------------------- |--------------------- |------ |------------:|----------:|----------:|------:|--------:|-------:|----------:|------------:|
| Execute | Core80 | Release | BIGINT | 100 | 16,219.7 us | 140.65 us | 131.57 us | 1.00 | 0.01 | - | 306.16 KB | 1.01 |
| Execute | NuGet80 | ReleaseNuGet | BIGINT | 100 | 16,283.9 us | 29.35 us | 26.01 us | 1.00 | 0.00 | - | 301.65 KB | 1.00 |
| | | | | | | | | | | | | |
| Fetch | Core80 | Release | BIGINT | 100 | 372.1 us | 2.91 us | 2.43 us | 1.00 | 0.01 | 2.9297 | 55.66 KB | 1.09 |
| Fetch | NuGet80 | ReleaseNuGet | BIGINT | 100 | 371.3 us | 2.47 us | 2.19 us | 1.00 | 0.01 | 2.9297 | 51.15 KB | 1.00 |
| | | | | | | | | | | | | |
| Execute | Core80 | Release | VARCH(...) UTF8 [30] | 100 | 18,021.3 us | 352.58 us | 482.62 us | 1.00 | 0.03 | - | 310.1 KB | 1.01 |
| Execute | NuGet80 | ReleaseNuGet | VARCH(...) UTF8 [30] | 100 | 18,071.4 us | 352.48 us | 377.14 us | 1.00 | 0.03 | - | 305.64 KB | 1.00 |
| | | | | | | | | | | | | |
| Fetch | Core80 | Release | VARCH(...) UTF8 [30] | 100 | 416.7 us | 6.50 us | 5.43 us | 1.01 | 0.03 | 2.9297 | 59.54 KB | 1.08 |
| Fetch | NuGet80 | ReleaseNuGet | VARCH(...) UTF8 [30] | 100 | 413.6 us | 8.09 us | 10.52 us | 1.00 | 0.03 | 2.9297 | 55.02 KB | 1.00 |
// * Hints *
Outliers
CommandBenchmark.Execute: NuGet80 -> 1 outlier was removed (17.09 ms)
CommandBenchmark.Fetch: Core80 -> 2 outliers were removed (383.60 us, 403.48 us)
CommandBenchmark.Fetch: NuGet80 -> 1 outlier was removed (389.54 us)
CommandBenchmark.Execute: Core80 -> 1 outlier was removed, 3 outliers were detected (16.46 ms, 16.87 ms, 19.28 ms)
CommandBenchmark.Fetch: Core80 -> 2 outliers were removed (442.98 us, 447.90 us)
// * Legends *
DataType : Value of the 'DataType' parameter
Count : Value of the 'Count' parameter
Mean : Arithmetic mean of all measurements
Error : Half of 99.9% confidence interval
StdDev : Standard deviation of all measurements
Ratio : Mean of the ratio distribution ([Current]/[Baseline])
RatioSD : Standard deviation of the ratio distribution ([Current]/[Baseline])
Gen0 : GC Generation 0 collects per 1000 operations
Allocated : Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)
Alloc Ratio : Allocated memory ratio distribution ([Current]/[Baseline])
1 us : 1 Microsecond (0.000001 sec)
// * Diagnostic Output - MemoryDiagnoser *
// ***** BenchmarkRunner: End *****
Run time: 00:01:41 (101.22 sec), executed benchmarks: 8
Global total time: 00:01:51 (111.73 sec), executed benchmarks: 8
@fdcastel - Interesting, and thanks! Taking a look now. I had the same thought the Perf project toolchain was out of date.
Okay, I see how the performance project works now with the conditional project/package reference, and with the latest Nuget release being the baseline. That is way better approach than the way I was attempting to compare the the results historically.
@fdcastel - Thank you very much for showing me how the performance project works. I just want try and confirm we haven't regressed in someway with this change.