FrameworkBenchmarks icon indicating copy to clipboard operation
FrameworkBenchmarks copied to clipboard

reported results are not feasible

Open robaho opened this issue 1 year ago • 6 comments

The vast majority of the plaintext results are not possible on 10 Gbit ethernet.

The maximum transfer speed of 10gbit ethernet is 1,250 megabytes per second.

The minimum valid http response for plaintext is 142 bytes, the headers are 129 bytes, and the payload 13 bytes.

HTTP/1.1 200 OK
Server: httpserver-robaho
Date: Tue, 16 Jul 2024 14:35:23 GMT
Content-type: text/plain
Content-length: 13 

You could reduce it a bit will a smaller server name.

so, the top result of 28,733,734 by mrhttp is not possible, as this would equate to:

28733734 * 142 / (1024*1024) = 3891 megabytes per second

The approximate theoretical maximum requests per second between a single client and single server machine on 10gbit ethernet is 9230422 requests per second - which would mean everything above 70th place is impossible. This also assumes that the packet rate is not exceeded (i.e. that every ethernet frame is filled to capacity).

The only way this would be possible is using multiple coordinated client machines with dedicated ethernet connections to multiple server machines - but the latest environment documentation does not describe this as being the case.

Can you provide more details on how these numbers are achieved? Thanks.

robaho avatar Jul 16 '24 14:07 robaho

I think I see the issue. I don't think the website is working properly. I think it is showing cloud results for the physical as they are the same.

image image

robaho avatar Jul 16 '24 15:07 robaho

Strangely though, the official benchmarks do not show any results for Cloud for plaintext or json serialization.

robaho avatar Jul 16 '24 15:07 robaho

There is currently no cloud environment.

The current environment is 40gbit.

Have to update the docs.

NateBrady23 avatar Jul 16 '24 15:07 NateBrady23

There is currently no cloud environment.

The current environment is 40gbit.

Have to update the docs.

When you click cloud, the results should probably be empty then to not confuse people.

robaho avatar Jul 16 '24 16:07 robaho

Can you provide more details on how these numbers are achieved? Thanks.

There are many explanations here: https://github.com/TechEmpower/FrameworkBenchmarks/issues/7402

Here I repeated the results on 99% identical hardware for the libreactor framework: https://github.com/TechEmpower/FrameworkBenchmarks/issues/7402#issuecomment-1482918369

There I also showed the throughput when using a 25Gbit network: 21387110 req/sec

remittor avatar Jul 18 '24 07:07 remittor

Can you provide more details on how these numbers are achieved? Thanks.

There are many explanations here: #7402

Here I repeated the results on 99% identical hardware for the libreactor framework: #7402 (comment)

There I also showed the throughput when using a 25Gbit network: 21387110 req/sec

Understood. The docs stated it was 10gbit Ethernet which is what made them infeasible.

robaho avatar Jul 18 '24 11:07 robaho