camel-quarkus icon indicating copy to clipboard operation
camel-quarkus copied to clipboard

Published 20 hours ago • verified publisher icon apache

perf-regression: collect report at each staging release

Open aldettinger opened this issue 2 years ago • 30 comments

The idea is to start collecting performance regression report at each staging release. From there, we would learn whether:

  • [ ] the report generation can be run on distinct machines, operating systems
  • [ ] reports looks consistent across versions, machines, operating systems
  • [x] the default duration of 10 minutes is a good trade-off to quickly spot potential regressions
  • [x] the current scenario provide the best value
  • [x] it would be valuable to reject a release with -5% regression detected

aldettinger avatar Aug 18 '22 09:08 aldettinger 

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                2.11.0  |       10m  |  15334.07 req/s [+0.00%]  |    8850.31 req/s [+0.00%]  |      OK  |
                2.12.0  |       10m  |  15343.82 req/s [+0.06%]  |    8976.51 req/s [+1.43%]  |      OK  |

aldettinger avatar Sep 01 '22 19:09 aldettinger 

Camel Quarkus Throughput Performance Increase Compared to Previous Version                
Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
               2.12.0  |       10m  |  21492.67 req/s [+0.00%]  |   11516.03 req/s [+0.00%]  |      OK  |
               2.13.0  |       10m  |  21299.82 req/s [-0.90%]  |   11828.55 req/s [+2.71%]  |      OK  |

zbendhiba avatar Sep 23 '22 14:09 zbendhiba 

Camel Quarkus Throughput Performance Increase Compared to Previous Version                
Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                2.13.0  |       10m  |  38872.21 req/s [+0.00%]  |   22154.66 req/s [+0.00%]  |      OK  |
                2.15.0  |       10m  |  38605.46 req/s [-0.69%]  |   22438.48 req/s [+1.28%]  |      OK  |

aldettinger avatar Jan 10 '23 18:01 aldettinger 

 Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                2.15.0  |       10m  |  38922.37 req/s [+0.00%]  |   22605.97 req/s [+0.00%]  |      OK  |
                2.16.0  |       10m  |  39248.04 req/s [+0.84%]  |   22543.42 req/s [-0.28%]  |      OK  |

aldettinger avatar Jan 19 '23 15:01 aldettinger 

 Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                2.13.1  |       10m  |  38888.74 req/s [+0.00%]  |   22779.41 req/s [+0.00%]  |      OK  |
                2.13.2  |       10m  |  38918.29 req/s [+0.08%]  |   23068.36 req/s [+1.27%]  |      OK  |

aldettinger avatar Jan 26 '23 20:01 aldettinger 

 Camel Quarkus Throughput Performance Increase Compared to Previous Version                             
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |               Status               |
------------------------------------------------------------------------------------------------------------------------------------
                2.16.0  |       10m  |  23394.75 req/s [+0.00%]  |   13429.86 req/s [+0.00%]  |                                OK  |
              3.0.0-M1  |       10m  |  23976.32 req/s [+2.49%]  |   12576.25 req/s [-6.36%]  |  Potential performance regression  |
2023-03-20 20:33:52,502 INFO  [io.quarkus] (main) perf-regression stopped in 0.006s

zbendhiba avatar Mar 20 '23 19:03 zbendhiba 

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                2.13.2  |       10m  |  39515.19 req/s [+0.00%]  |   23098.37 req/s [+0.00%]  |      OK  |
                2.13.3  |       10m  |  39745.16 req/s [+0.58%]  |   22774.74 req/s [-1.40%]  |      OK  |

aldettinger avatar Apr 14 '23 11:04 aldettinger 

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
              3.0.0-M1  |       10m  |  25477.00 req/s [+0.00%]  |   13664.18 req/s [+0.00%]  |      OK  |
              3.0.0-M2  |       10m  |  27129.07 req/s [+6.48%]  |   14002.18 req/s [+2.47%]  |      OK  |

ppalaga avatar Jun 01 '23 10:06 ppalaga

Comparison between version 2.16 and the latest 3.0.0-M2 to assess improvements since the M1 regression

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                2.16.0  |       10m  |  22896.22 req/s [+0.00%]  |   13133.90 req/s [+0.00%]  |      OK  |
              3.0.0-M2  |       10m  |  23067.02 req/s [+0.75%]  |   13404.82 req/s [+2.06%]  |      OK  |

zbendhiba avatar Jun 15 '23 18:06 zbendhiba 

               Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |  JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
---------------------------------------------------------------------------------------------------------
              3.0.0-M2  |       10m  |  9569.35 req/s [+0.00%]  |    6022.07 req/s [+0.00%]  |      OK  |
             3.0.0-RC1  |       10m  |  9091.22 req/s [-5.00%]  |    6312.80 req/s [+4.83%]  |      OK  |

ppalaga avatar Jul 04 '23 11:07 ppalaga 

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
             3.0.0-RC1  |       10m  |  39443.07 req/s [+0.00%]  |   22666.68 req/s [+0.00%]  |      OK  |
             3.0.0-RC2  |       10m  |  39347.05 req/s [-0.24%]  |   22559.33 req/s [-0.47%]  |      OK  |

aldettinger avatar Aug 02 '23 11:08 aldettinger 

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
             3.0.0-RC2  |       10m  |  40384.40 req/s [+0.00%]  |   22582.53 req/s [+0.00%]  |      OK  |
                 3.2.0  |       10m  |  39638.48 req/s [-1.85%]  |   22462.29 req/s [-0.53%]  |      OK  |

aldettinger avatar Oct 03 '23 20:10 aldettinger 

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.2.0  |       10m  |  39873.77 req/s [+0.00%]  |   22604.50 req/s [+0.00%]  |      OK  |
                 3.4.0  |       10m  |  39788.82 req/s [-0.21%]  |   22057.67 req/s [-2.42%]  |      OK  |

aldettinger avatar Oct 03 '23 20:10 aldettinger 

 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.2.0  |       10m  |  19495.67 req/s [+0.00%]  |   11577.53 req/s [+0.00%]  |      OK  |
                 3.2.1  |       10m  |  19257.06 req/s [-1.22%]  |   11733.54 req/s [+1.35%]  |      OK  |

ppalaga avatar Oct 06 '23 16:10 ppalaga 

                             Camel Quarkus Throughput Performance Increase Compared to Previous Version                              
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]    |  Native req/s [%increase]  |               Status               |
-------------------------------------------------------------------------------------------------------------------------------------
                 3.4.0  |       10m  |   20250.91 req/s [+0.00%]  |   12564.35 req/s [+0.00%]  |                                OK  |
                 3.5.0  |       10m  |  15222.80 req/s [-24.83%]  |   9273.02 req/s [-26.20%]  |  Potential performance regression  |

jamesnetherton avatar Oct 23 '23 15:10 jamesnetherton 

                             Camel Quarkus Throughput Performance Increase Compared to Previous Version                              
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]    |  Native req/s [%increase]  |               Status               |
-------------------------------------------------------------------------------------------------------------------------------------
                 3.4.0  |       10m  |   20250.91 req/s [+0.00%]  |   12564.35 req/s [+0.00%]  |                                OK  |
                 3.5.0  |       10m  |  15222.80 req/s [-24.83%]  |   9273.02 req/s [-26.20%]  |  Potential performance regression  |

Woops, this looks pretty bad. And it's confirmed on another machine, with 10m and 1h long runs. Let's deal with this in #5452.

aldettinger avatar Oct 23 '23 16:10 aldettinger 

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.2.1  |       10m  |  19241.25 req/s [+0.00%]  |   12220.36 req/s [+0.00%]  |      OK  |
                 3.2.2  |       10m  |  19023.79 req/s [-1.13%]  |   11801.31 req/s [-3.43%]  |      OK  |

jamesnetherton avatar Nov 03 '23 16:11 jamesnetherton 

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]    |  Native req/s [%increase]  |  Status  |
-----------------------------------------------------------------------------------------------------------
                 3.5.0  |       10m  |   10184.32 req/s [+0.00%]  |    6655.12 req/s [+0.00%]  |      OK  |
                 3.6.0  |       10m  |  11328.00 req/s [+11.23%]  |   7618.24 req/s [+14.47%]  |      OK  |

jamesnetherton avatar Nov 27 '23 08:11 jamesnetherton 

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.2.2  |       10m  |  26745.20 req/s [+0.00%]  |   16353.16 req/s [+0.00%]  |      OK  |
                 3.2.3  |       10m  |  25663.29 req/s [-4.05%]  |   15666.90 req/s [-4.20%]  |      OK  |

zhfeng avatar Dec 06 '23 13:12 zhfeng 

                Camel Quarkus Throughput Performance Increase Compared to Previous Version
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.6.0  |       10m  |  20520.59 req/s [+0.00%]  |   11885.86 req/s [+0.00%]  |      OK  |
                 3.7.0  |       10m  |  20577.29 req/s [+0.28%]  |   11885.30 req/s [-0.00%]  |      OK  |

jamesnetherton avatar Jan 26 '24 10:01 jamesnetherton 

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.7.0  |       10m  |  33532.36 req/s [+0.00%]  |   17749.33 req/s [+0.00%]  |      OK  |
                 3.8.0  |       10m  |  32340.26 req/s [-3.56%]  |   17498.96 req/s [-1.41%]  |      OK  |

aldettinger avatar Feb 22 '24 08:02 aldettinger 

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.8.0  |       10m  |  19217.32 req/s [+0.00%]  |   11275.71 req/s [+0.00%]  |      OK  |
                 3.8.1  |       10m  |  19457.29 req/s [+1.25%]  |   11112.60 req/s [-1.45%]  |      OK  |

jamesnetherton avatar Mar 18 '24 08:03 jamesnetherton 

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.8.0  |       10m  |  32583.61 req/s [+0.00%]  |   17899.62 req/s [+0.00%]  |      OK  |
                 3.9.0  |       10m  |  32610.64 req/s [+0.08%]  |   17575.19 req/s [-1.81%]  |      OK  |

aldettinger avatar Mar 22 '24 08:03 aldettinger 

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.9.0  |       10m  |  19907.84 req/s [+0.00%]  |   11237.27 req/s [+0.00%]  |      OK  |
                3.10.0  |       10m  |  19803.77 req/s [-0.52%]  |   11447.46 req/s [+1.87%]  |      OK  |

jamesnetherton avatar Apr 25 '24 09:04 jamesnetherton 

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.8.1  |       10m  |  19543.29 req/s [+0.00%]  |   11300.57 req/s [+0.00%]  |      OK  |
                 3.8.2  |       10m  |  19289.10 req/s [-1.30%]  |   11163.94 req/s [-1.21%]  |      OK  |

jamesnetherton avatar May 03 '24 13:05 jamesnetherton 

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                3.10.0  |       10m  |  31382.21 req/s [+0.00%]  |   16818.13 req/s [+0.00%]  |      OK  |
                3.11.0  |       10m  |  30932.51 req/s [-1.43%]  |   17040.53 req/s [+1.32%]  |      OK  |

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                3.11.0  |       10m  |  31612.49 req/s [+0.00%]  |   16867.78 req/s [+0.00%]  |      OK  |
                3.12.0  |       10m  |  30904.69 req/s [-2.24%]  |   17089.98 req/s [+1.32%]  |      OK  |

aldettinger avatar Jun 20 '24 11:06 aldettinger 

                Camel Quarkus Throughput Performance Increase Compared to Previous Version                
 Camel Quarkus version  |  Duration  |   JVM req/s [%increase]   |  Native req/s [%increase]  |  Status  |
----------------------------------------------------------------------------------------------------------
                 3.8.2  |       10m  |  19512.99 req/s [+0.00%]  |   11358.99 req/s [+0.00%]  |      OK  |
                 3.8.3  |       10m  |  20256.12 req/s [+3.81%]  |   11189.47 req/s [-1.49%]  |      OK  |

jamesnetherton avatar Jul 03 '24 13:07 jamesnetherton