FrameworkBenchmarks icon indicating copy to clipboard operation
FrameworkBenchmarks copied to clipboard
verified publisher icon TechEmpower

List of 50 longest TFB tests

Open remittor opened this issue 2 years ago • 6 comments

TFB results of 2023-04-06: https://www.techempower.com/benchmarks/#section=test&runid=ab65d6b5-8efc-4a65-9c5c-c31667418528

Full stat of tests: https://tfb-status.techempower.com/raw/results.2023-04-06-08-48-50-340.zip (56MB)

Parsed stats: Format: total_test_time (build_time + verify_time) test_name

$ python3 tfb_res.py -f results.2023-04-06-08-48-50-340.zip
Number of frameworks: 295
Number of tests: 767
Total time: 6 days, 15:26:25 (included 12.78 hours of sleep)
Total time: 146.66 + 12.78 = 159.44 hours
Total build  time: 27.65 hours
Total verify time: 114.96 hours
Verify time avg: (114.96 * 60) / 767 = 8.99 min
Number of full tests: 312
Full tests verify time avg: (68.03 * 60) / 312 = 13.08 min

0:39:10 (0:32:35 + 0:06:17)  wizzardo-inline
0:33:28 (0:02:48 + 0:30:27)  viz-diesel
0:29:23 (0:16:10 + 0:12:59)  warp
0:29:01 (0:15:53 + 0:12:58)  drogon
0:28:41 (0:15:19 + 0:12:59)  warp-mysql-haskell
0:28:30 (0:15:17 + 0:12:59)  warp-hasql
0:27:06 (0:14:42 + 0:11:12)  spliffy-mongodb
0:26:34 (0:22:28 + 0:03:59)  servant
0:26:19 (0:21:32 + 0:04:33)  servant-psql-simple
0:26:11 (0:14:44 + 0:11:12)  spliffy-postgres
0:26:10 (0:14:37 + 0:11:11)  spliffy-mysql
0:25:18 (0:00:36 + 0:24:15)  hapi-nginx
0:25:12 (0:16:02 + 0:08:57)  aspcore-aot-ado-pg
0:25:06 (0:08:42 + 0:15:15)  treefrog-mongodb
0:25:03 (0:15:52 + 0:08:58)  drogon-core
0:24:52 (0:09:23 + 0:15:16)  treefrog
0:24:30 (0:14:53 + 0:09:19)  spring-webflux-rxjdbc
0:24:21 (0:08:43 + 0:15:16)  treefrog-mysql
0:24:16 (0:08:48 + 0:15:15)  treefrog-epoll
0:24:15 (0:01:18 + 0:22:42)  rack-sequel-postgres-passenger-mri
0:22:39 (0:01:17 + 0:20:56)  rack-sequel-passenger-mri
0:22:35 (0:08:49 + 0:13:34)  userver
0:22:22 (0:13:11 + 0:08:58)  ntex-db-astd
0:22:07 (0:08:44 + 0:13:01)  phalcon
0:22:00 (0:06:40 + 0:15:10)  anansi
0:21:49 (0:10:27 + 0:11:02)  cppcms
0:21:40 (0:10:40 + 0:10:45)  cppcms-postgres
0:21:31 (0:10:10 + 0:11:10)  appmpower-ado-pg
0:21:24 (0:06:02 + 0:15:09)  userver-bare
0:19:52 (0:06:31 + 0:13:00)  lumen-workerman
0:19:50 (0:08:41 + 0:10:46)  phalcon-micro
0:19:46 (0:06:23 + 0:13:01)  lumen-laravel-s
0:19:34 (0:06:15 + 0:12:59)  mixphp-swoole-mysql
0:19:30 (0:06:18 + 0:12:58)  ubiquity-swoole
0:19:27 (0:03:55 + 0:15:09)  elixir-plug-ecto
0:19:27 (0:08:14 + 0:11:03)  ihp
0:19:21 (0:05:57 + 0:13:00)  laravel-swoole
0:19:21 (0:06:08 + 0:13:00)  lucky
0:19:20 (0:05:56 + 0:13:01)  laravel-laravel-s
0:19:02 (0:03:41 + 0:15:09)  h2o
0:18:57 (0:14:51 + 0:03:59)  spliffy
0:18:56 (0:05:39 + 0:13:01)  ningle
0:18:55 (0:05:35 + 0:12:58)  one-no-coroutine
0:18:40 (0:14:36 + 0:03:59)  0http
0:18:39 (0:05:18 + 0:12:58)  swoole
0:18:38 (0:05:15 + 0:13:00)  lumen
0:18:38 (0:05:20 + 0:13:01)  woo
0:18:36 (0:03:03 + 0:15:10)  cfml
0:18:28 (0:05:06 + 0:13:00)  lumen-swoole
0:18:28 (0:05:18 + 0:12:58)  rocket

Parser source code: https://gist.github.com/remittor/38ef89e47c18f6ec2a2095bc7390cdf3

remittor avatar Apr 06 '23 15:04 remittor

Parsed stats for frameworks: Format: total_test_time (build_time + verify_time) framework_name

$ python3 tfb_res.py --fw -f results.2023-04-06-08-48-50-340.zip
Number of frameworks: 295
Number of tests: 767
Total time: 6 days, 15:26:25 (included 12.78 hours of sleep)
Total time: 146.66 + 12.78 = 159.44 hours
Total build  time: 27.65 hours
Total verify time: 114.96 hours
Verify time avg: (114.96 * 60) / 767 = 8.99 min
Number of full tests: 312
Full tests verify time avg: (68.03 * 60) / 312 = 13.08 min

3:32:39 (0:16:58 + 3:12:51)  http4k
2:52:36 (0:09:14 + 2:40:22)  officefloor
2:25:48 (0:39:08 + 1:42:56)  ubiquity
2:17:17 (0:54:02 + 1:17:45)  cutelyst
2:16:51 (0:23:52 + 1:49:07)  php
2:12:59 (0:08:39 + 2:01:38)  rack-sequel
2:04:03 (0:17:55 + 1:43:27)  falcon
2:02:45 (0:14:35 + 1:46:33)  hexagon
1:57:36 (0:28:35 + 1:24:59)  micronaut   ERROR: "Problem starting micronaut-graalvm"
1:51:30 (0:11:01 + 1:35:05)  play2
1:51:19 (0:08:19 + 1:40:15)  sinatra-sequel
1:48:03 (0:08:58 + 1:33:13)  pippo
1:47:13 (0:07:50 + 1:36:32)  fastapi
1:43:48 (0:17:06 + 1:24:16)  imi
1:42:50 (0:07:54 + 1:31:18)  nestjs
1:41:11 (0:12:22 + 1:26:59)  ktor   ERROR: "Problem starting ktor-exposed-dao"
1:40:52 (0:31:50 + 1:05:01)  spring
1:38:35 (0:35:36 + 1:01:02)  treefrog
1:38:24 (0:58:54 + 0:37:34)  spliffy
1:31:33 (0:45:54 + 0:42:07)  axum
1:29:46 (0:02:07 + 1:25:01)  chi
1:28:21 (0:21:26 + 1:05:01)  laravel
1:26:45 (0:06:51 + 1:17:58)  roda-sequel
1:26:34 (0:46:46 + 0:38:57)  warp
1:20:54 (0:34:48 + 0:43:50)  actix
1:19:36 (0:07:30 + 1:10:09)  sinatra
1:16:44 (0:23:15 + 0:52:01)  lumen
1:14:00 (0:02:25 + 1:09:07)  ffead-cpp
1:09:10 (0:01:16 + 1:04:12)  nodejs
1:06:18 (0:28:01 + 0:37:30)  appmpower
1:05:01 (0:03:00 + 1:00:56)  jooby
1:04:46 (0:11:37 + 0:52:22)  viz
1:03:35 (0:14:23 + 0:47:55)  mixphp
0:59:41 (0:28:26 + 0:29:53)  ntex   ERROR: "Problem starting ntex-plt-astd"
0:59:11 (0:01:35 + 0:55:24)  hapi
0:56:06 (0:07:04 + 0:45:33)  vertx-web   ERROR: "Framework is not accepting requests from client machine"
0:54:50 (0:09:39 + 0:43:55)  kumbiaphp
0:54:04 (0:31:45 + 0:21:56)  drogon
0:54:00 (0:05:10 + 0:45:30)  lithium
0:52:53 (0:44:00 + 0:08:32)  servant
0:52:28 (0:02:45 + 0:48:44)  microdot
0:51:16 (0:13:51 + 0:36:06)  roa
0:49:57 (0:03:48 + 0:45:28)  redkale
0:47:44 (0:15:51 + 0:30:55)  swoole
0:47:43 (0:07:07 + 0:39:48)  xitca-web
0:46:54 (0:22:56 + 0:21:59)  vapor
0:46:31 (0:14:36 + 0:30:55)  php-ngx
0:46:13 (0:14:20 + 0:30:57)  openswoole
0:44:53 (0:05:02 + 0:38:57)  quarkus
0:44:28 (0:08:34 + 0:34:55)  slim

remittor avatar Apr 06 '23 18:04 remittor

Rust framework always much time on build docker images . Is taht possible to use docker muilt-build stage or shared a comon build image or other way to share the compile results?

chrislearn avatar Apr 06 '23 20:04 chrislearn

Let's take a look at this: 0:33:28 (0:02:48 + 0:30:27) viz-diesel Verify time = 30.5 min (avg time = 13 min)

Let's look at the details for test fortunes: https://tfb-status.techempower.com/unzip/results.2023-04-06-08-48-50-340.zip/results/20230330235259/viz-diesel/fortune

 Concurrency: 16 for fortune
 wrk -H 'Host: 10.0.0.1' -H 'Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 15 -c 16 --timeout 8 -t 16 "http://10.0.0.1:8080/fortunes"
---------------------------------------------------------
unable to connect to 10.0.0.1:8080 Connection timed out
STARTTIME 1680761646
ENDTIME 1680761776

All fortunes tests failed and took a lot of time: 776 - 646 = 130 sec (2.17 min) 2.17 min * 8 tests = 17.33 min

Checking the result by the time the file was created stat.txt and raw.txt: 1680751642 - 1680750576 = 1066 sec = 17.76 min

Ideally, each test should take 15 seconds. So the ideal total time for fortune test should be: 15 * 8 = 120 seconds (2 min).

But viz-diesel fortunes test somehow managed to pass the test:

--------------------------------------------------------------------------------
VERIFYING FORTUNE
--------------------------------------------------------------------------------
   PASS for http://10.0.0.1:8080/fortunes
     Executed queries: 512/512
   PASS for http://10.0.0.1:8080/fortunes
     Rows read: 6077/6144

Test pre-verification time: 30.5 min - 17.76 min = 12.74 min

Verify test in CI: https://github.com/TechEmpower/FrameworkBenchmarks/actions/runs/4554118663/jobs/8031564280#step:9:876

UPDATE Full stat for this test:

$ python3 tfb_res.py -f results.2023-04-06-08-48-50-340.zip -t viz-diesel
Number of frameworks: 295
Number of tests: 767
Total time: 6 days, 15:26:25 (included 12.78 hours of sleep)
Total time: 146.66 + 12.78 = 159.44 hours
Total build  time: 27.65 hours
Total verify time: 114.96 hours
Verify time avg: (114.96 * 60) / 767 = 8.99 min
Number of full tests: 316
Full tests verify time avg: (68.86 * 60) / 316 = 13.07 min

0:33:28 (0:02:48 + 0:30:27)  viz-diesel

                  verify    test
             db : 0:00:04   0:02:12
          query : 0:00:16   0:01:54
         update : 0:00:22   0:07:42
        fortune : 0:00:04   0:17:46

          TOTAL : 0:00:46   0:29:34

Total verify and test time: 0:30:20

remittor avatar Apr 07 '23 08:04 remittor

Let's take a look at this: 0:24:15 (0:01:18 + 0:22:42) rack-sequel-postgres-passenger-mri Verify time = 22.7 min (avg time = 13 min)

Let's look at the details for test fortunes: https://tfb-status.techempower.com/unzip/results.2023-04-06-08-48-50-340.zip/results/20230330235259/rack-sequel-postgres-passenger-mri

Full stat for this test:

$ python3 tfb_res.py -f results.2023-04-06-08-48-50-340.zip -t rack-sequel-postgres-passenger-mri
Number of frameworks: 295
Number of tests: 767
Total time: 6 days, 15:26:25 (included 12.78 hours of sleep)
Total time: 146.66 + 12.78 = 159.44 hours
Total build  time: 27.65 hours
Total verify time: 114.96 hours
Verify time avg: (114.96 * 60) / 767 = 8.99 min
Number of full tests: 316
Full tests verify time avg: (68.86 * 60) / 316 = 13.07 min

0:24:15 (0:01:18 + 0:22:42)  rack-sequel-postgres-passenger-mri

                  verify    test
             db : 0:00:06   0:04:04
           json : 0:00:04   0:05:22
          query : 0:00:16   0:02:58
         update : 0:00:16   0:03:00
        fortune : 0:00:04   0:03:54
      plaintext : 0:00:04   0:02:24

          TOTAL : 0:00:50   0:21:42

Total verify and test time: 0:22:32

Look to /json/raw.txt : https://tfb-status.techempower.com/unzip/results.2023-04-06-08-48-50-340.zip/results/20230330235259/rack-sequel-postgres-passenger-mri/json/raw.txt

 Concurrency: 256 for json
 wrk -H 'Host: 10.0.0.1' -H 'Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 15 -c 256 --timeout 8 -t 28 "http://10.0.0.1:8080/json"
---------------------------------------------------------
Running 15s test @ http://10.0.0.1:8080/json
  28 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.12ms    1.02ms  11.65ms   79.87%
    Req/Sec     1.12k   159.09     1.26k    89.29%
  Latency Distribution
     50%    6.13ms
     75%    6.67ms
     90%    7.26ms
     99%    8.76ms
  3110 requests in 15.09s, 735.41KB read
Requests/sec:    206.16
Transfer/sec:     48.75KB
STARTTIME 1680629011
ENDTIME 1680629091

1680629091 - 1680629011 = 80 sec

remittor avatar Apr 07 '23 12:04 remittor

@remittor where did you get tfb_res.py? I can't find it in the repository.

I'm looking the way to output total stats for results of tests started in my local environment. Ideally if it will be possible to view a comparison between two runs. Is there such tool?

svetlyak40wt avatar May 14 '23 09:05 svetlyak40wt

@svetlyak40wt , look first post https://gist.github.com/remittor/38ef89e47c18f6ec2a2095bc7390cdf3

remittor avatar May 14 '23 10:05 remittor