FrameworkBenchmarks
FrameworkBenchmarks copied to clipboard
TechEmpower
Tests that need updating before next round
This is a living list of tests that need to be updated or removed before we record our next official round results. I'll do my best to ping the most recent maintainers as well as list the problem. Tests that aren't updated/fixed prior to the next round will be removed to free up resources and speed up the testing cycle.
Tests that are broken
C++/ffead-cpp Crystal/spider-gazelle Dart/hunt-dmd Go/chi-* Go/revel Haskell/servant-beam Haskell/spock
Tests that are failing query verification #5145
C++/cpoll_cppsp-postgres-raw C++/cpoll_cppsp-postgres-raw-threadpool C++/cpoll_cppsp-raw C++/cppcms C++/silicon C++/treefrog-hybrid Dart/hunt-http Elixir/phoenix
Tests that are not reading the id column in the queries or db tests #5167
D/hunt D/vibed
This list is still growing. I'm not even a 3rd of the way through my audit. Please feel free to contribute in the comments section.
I had already taken a little lead starting from the end of the Travis run:
Tests that are broken
- Swift/perfect-mongodb (Queries & Update tests)
- Swift/perfect-mysql
- Swift/perfect-postgresql
- Scala/play2-scala-anorm (Fortune test)
- Scala/akka-http
- Ruby/hanami
- Ruby/hanami-unicorn
- Ruby/rack
- Ruby/rack-falcon
- Ruby/rack-unicorn
- Python/vibora
- PHP/hhvm
- Perl/dancer
- Perl/kelp
- Perl/kelp-mongodb
- Perl/web-simple
- Nim/httpbeast
- Nim/jester
- Lua/octopus
- PHP/cakephp (Fortune test)
- Javascript/express-graphql-mysql (Update test)
- Java/spring-webflux
- Java/servlet-mysql
- Java/minijax
- Java/redkale-postgres (Db test in Travis)
Tests that are failing query verification https://github.com/TechEmpower/FrameworkBenchmarks/pull/5145
- TypeScript/typescript-rest (Update test ->no reading before update)
- Swift/kitura-mongodb (Update test)
- Scala/play2-scala-anorm-netty (Fortune test)
- Rust/rocket (Update test -> rows updated)
- Ruby/rails-postgresql (Queries test)
- Ruby/rails-unicorn
- Ruby/sinatra (Fortune test)
- Ruby/sinatra-postrges (Fortune test)
- Python/morepath (Query, Update tests)
- Python/pyramid (Update test)
- Python/pyramid-py2 (Update test)
- Python/quart (Update test)
- Python/spyne (Query, Update tests)
- Python/spyne-nginx-uwsgi (Db, Query, Update tests)
- Python/bottle (Db, update tests)
- Python/bottle-nginx-uwsgi
- Python/cherrypy
- PHP/imi (Update test)
- Perl/mojolicious (Query, Update tests)
- Kotlin/pronghorn (Db, Query, Update tests)
- Javascript/hapi (Query test)
- Javascript/koa (Update test)
- Javascript/express-postgres (Update test ->no reading before update)
- Javascript/express-graphql-postgres (Update test ->no reading before update)
- Javascript/express-chakra (Update test)
- Java/spring-webflux-mongo (Update test)
- Java/quarkus (Update test)
-
Java/officefloor (Query, Update tests) -> Hibernate pb? (see https://github.com/TechEmpower/FrameworkBenchmarks/pull/5145#issuecomment-544735202)
- officefloor-netty
- officefloor-rapidoid
- officefloor-spring_data
- Java/dropwizard (Update test)
- Java/dropwizard-jdbi-postgres (Db, Query, Fortune tests)
- Java/dropwizard-postgres(Db, Query, Update & Fortune tests)
- Groovy/hot-* (All db tests)
- C++/ulib (Update test, blocking for Travis on Query)
- C++/wt-postgres (Query, Update tests)
Tests that are not reading the id column in the queries or db tests https://github.com/TechEmpower/FrameworkBenchmarks/issues/5167
- Python/quart
- Python/aiohttp
- PHP/hhvm
All php frameworks are correct after PR #5192 for #5167.
The only that need the change is hhvm, but it is broken a long time ago.
@jcheron You marked frameworks that are correct. php, php-ngx, fat-fee, workerman, kumbiaphp, ...
They only omit the id in the update tests that is permited.
https://www.techempower.com/benchmarks/#section=test&runid=59583d86-f7b6-4e86-b86a-a7e8eebf89fb&hw=ph&test=fortune&l=zik073-v the last row cakephp - Did not complete and https://tfb-status.techempower.com/unzip/results.2019-10-28-04-17-30-832.zip/results/20191024072803/cakephp/fortune/raw.txt
But the run log is empty https://tfb-status.techempower.com/unzip/results.2019-10-28-04-17-30-832.zip/results/20191024072803/cakephp/run
That happens after update the cakephp version.
It looks good on travis, but the fortunes test always failed for the last 6 months in citrine and azure.
In azure: https://www.techempower.com/benchmarks/#section=test&runid=45c588f9-9968-4330-934f-bd39ed92442c&hw=ph&test=fortune&l=zik073-v
I think that we could create an issue for the php frameworks, to clean this issue. Created issue #5193
@nbrady-techempower ,Hey, I have updated drogon with the PR #5191 , I think you can remove drogon from the list, thanks.
@jcheron You commented in your PR that several Java frameworks using Hibernate were failing query verification, but they're not in your list in this thread. Why is that?
@michaelhixson I continue the pointing, starting from the bottom, I'm close to java...
Tests that are not reading the id column in the queries or db tests #5167
C++/cpoll_cppsp (all variants) C++/ulib Crystal/crystal Crystal/kemal Crystal/raze D/hunt (all variants) Perl/mojolicious Perl/plack Python/aiohttp Python/quart Racket/racket