Add a mixed test. Question open ?

[joanhey]

Actually a lot of devs say that this bench is too synthetic !! Almost all benchs are so.

Is it a question for the all devs helping with this benchmark !! To many tests of this bench test use the JIT help, that is OK. But in a realistic server will be so ???

I open a new test for discussion !!

The new test will use exactly all the same tests, but with the same test need to run all the tests, so the JIT will be less important. In a real server, we use different routes to manage it. But we have that already. We want to see a bench test with:

• 1 db
• 1 plaintext
• 1 fortune
• 1 json
• ...

In the same order for all the frameworks, and later we'll see a more realistic bench.

Perhaps I don't explained perfect, but I think that we need to reflexioned . 1 test each time, with the same order, to kill the JIT for only 1 test

We don't need to change any code to benchmark, we have all the tests, only join in 1 test.

It's an open question!!

PD: any permutation that don't use all the tests in the same bench will be excluded. EDIT: any framework will use pipeline in all requests ??

[NateBrady23]

I understand what you're getting at @joanhey. I'd be curious what other opinions are here. I may be able to spare somebody from R&D to add a future test type at some point. Love to hear people's thoughts on this.

[Kaliumhexacyanoferrat]

To address the "too synthetic" issue, you might want to add some HTTP server compliance check as well. There are quite a few frameworks that I would not consider a HTTP server or HTTP server framework but just very specific response generators to pass the benchmark. And they are of course fast as they do not need to handle all of the cases a regular HTTP server has to. So maybe there is a suite / command line utility that can be used to verify a HTTP server on a basic level - or we could spin off this project, similar what Acid did with HTML. As a result and a soft approach, failing frameworks could be marked as non-realistic, combined with a default filter on realistic tests for official rounds.

See https://x.com/uNetworkingAB/status/1811425564764610926 - I must admit I can understand the frustration there.

[volyrique]

This proposal sounds good to me. I don't think that it will have that much of an effect on JIT compilation, since it often works on individual functions/methods instead of traces (so the effect will probably be limited to the routing implementation), but the suggestion is reasonable otherwise, given that it is a somewhat more realistic case.

[Dreamsorcerer]

To highlight some things that don't appear on these benchmarks, but probably have a bigger impact on real-world usage, here are some things we've fixed recently on performance:

• Routing performance. Lots of routes (i.e. endpoints), routes with variables (maybe regex too), nested routes etc.. We've recently improved our routing performance, so for an app with 5000 endpoints the routing has been reduced from worst case of over 2 seconds (yes, really) to about 2ms (which is only 20% slower than the previous best case).
• Websockets. For applications like homeassistant with a single user, the endpoint performance is irrelevant, but devices on the network can flood the server with lots of websocket messages, therefore being able to parse websocket messages quickly is important.
• Middlewares. Many frameworks support middlewares to modify behaviour of all requests. With several middlewares involved, this can significantly affect performance. We tweaked our code to reduce the overhead of middlewares.