Help us select frameworks for measuring hardware performance (TPR)

[bhauer]

We are presently working on enhancing the TFB toolset to yield a measurement of the performance of the hardware environment (in addition to our measurement of specific frameworks). This is basically using our library of web-application framework tests to provide something of potential interest to a new audience of people measuring hardware performance.

We're tentatively calling this a TechEmpower Performance Rating. See the wiki article about TPR for more detail.

This issue is soliciting community feedback generally, but also specifically about which frameworks you feel should be used to measure hardware performance. We want the list to be a relatively small subset (10 or so) so that the total execution time for a hardware test is significantly less than our current ~100 hours of total execution time for the full suite.

The tentative list I am proposing is:

• ASP.NET Core / C#
• Django / Python
• Express / JavaScript
• Laravel / PHP
• Phoenix / Elixir
• Rails / Ruby
• Spring / Java/JVM
• ?? / C++ or C
• ?? / Go
• ?? / Rust

I'd really appreciate input on what to select for the last three. But if you have input on the others, please share.

Also, if you have ideas or thoughts about TPR in general, please share those as well. Thank you!

[NateBrady23]

@joanhey @jcheron Thoughts on the PHP framework(s)?

[jcheron]

For PHP, I would tend to say Laravel, given its audience, but I don't have a clear appreciation of the implications on the notion of TPR. There is a fracture today between php frameworks using asynchronous technologies and the others, and I don't know if that matters.

[bhauer]

Thanks for the input, @jcheron!

For the purposes of this, a large audience is probably a strong argument in Laravel's favor. We don't necessarily want the newest or fastest framework for this measurement. We'd rather choose something that is very popular and representative of a large number of applications in the real world.

[gotzmann]

PHP with Laravel or Symfony is must have!

[bhauer]

Sounds like Laravel is the right selection for PHP. I am going to edit the Wiki and the initial comment here to reflect that.

[volyrique]

If the requirement to implement all tests is strict, there is only one C choice, h2o, and only 3 C++ choices - cppcms, ffead-cpp, and ulib. Actually the implementations of the cached queries test type use C or C++ quite disproportionately - 6 out of 14 (that managed to complete).

Having the ability to reproduce the results for a particular round would require making another pass through the build configurations, Dockerfiles, scripts, etc. to make sure that all dependencies are really locked down (as much as possible). In particular, some of the database issues I have highlighted in #5185 need to be fixed.

[n-stefan]

It could be helpful to somehow highlight these exact frameworks so that it becomes very easy to see, on the results web site, how the most widely used frameworks (per language) compare to each other; maybe by also adding a filter option.

[bhauer]

@n-stefan We will definitely include an indicator next to the frameworks that are used to produce a hardware score. It will be a little badge icon next to the framework name.

[n-stefan]

@bhauer Great :)

[zloster]

The JVM world is very "difficult" - a lot of choice but nothing you really need. Some points:

• The current JVM/Spring implementation is not using Hibernate/SpringJPA. It is very hard to find an application without these in the database access layer. The performance figures were very different when Hibernate was utilised;
• There are 4 (four) mature implementations (Servlet, Tapestry, Wicket, Wildfly EE) that use "canonical" and simple implementation. Also no fancy "optimisations" like the single mega UPDATE query. But all of these are old technologies and development of new applications for them is highly unlikely. Moving such old applications to a new (containerized) environments is more likely to happen. But probably the application performance will be addressed on a moaning basis - if someone complains enough about the performance it will be addressed with some more resources.
• There are also Java/gemini and Java/undertow implementations. TechEmpower is putting some effort there. But Gemini is a Servlet application and Undertow project will not be the same after the IBM acquisition of RedHat.
• Also as the case with PHP the async/reactive fragmentation is too big. Every major vendor have it's own (Spring WebFlux - Pivotal/VMware, Quarkus - RedHat/IBM, Helidon - Oracle, ServiceTalk - Apple). Also Oracle moved a lot of stuff including Grizzly to the Eclipse Fondation.

IMO WildflyEE seems to represent most faithfully the old/current world. Netty is used in all of the fancy shiny new stuff from the vendors (Spring WebFlux, Quarkus, Helidon, ServiceTalk). If the test implementations of the Undertow project are translated to use Netty the result will be a good representation of the fancy shiny new stuff*.

[bhauer]

@zloster Thanks for the insight on the JVM situation.

I had not realized Undertow was switching to a Netty foundation. Isn't Undertow, in turn, the underpinnings for Wildfly?

Thoughts on DropWizard? Is that still fairly well regarded?

Regardless, unless there are more opinions on the JVM front, I feel comfortable with using WildflyEE for this purpose.

[zloster]

@bhauer Yes, Undertow is a part of the Wildfly's foundations. Some recent stack-traces for confirmation here and here. Bugfixes are promised for Undertow 2.x branch. Wildfly and undertow test implementations are using this version currently.

About Dropwizard: I've committed some of the code for the test implementations and I definitely like the Dropwizard's approach. I'm not sure if my opinion should be taken in account. Dropwizard represents a niche. A lot of frameworks/libraries took inspiration from it. And it's not a framework in the general sense. It is several libraries and several abstractions to glue them. The abstractions stood the test of time - take a look at the commit history of Grasshopper and Signal Server. And if you need some more functionality it is fairly easy - the modules directory and a recent batch of new modules.

But again choosing one framework is very difficult. Even more difficult will be defending the choice. If the task is tweaked to be "choose one framework for each language that is the most interesting from technical perspective" - I'll definitely would say Java/proteus. The idea to compile a JAX-RS annotated service class to Undertow handler gives very interesting opportunities. If it is possible for Undertow than it should be possible for Jetty and Netty. Something like JavaServerPages for RESTful webservices. But it doesn't have all the testcases implemented and needs a dependencies update. Update: To have more complete view on the idea with the compilation. I've just found out that https://github.com/buremba/netty-rest exist. It claims JAX-RS compilation to Netty code. It's first commit here on GitHub predates Proteus with 2 years (2015 vs 2017). So implementations for Grizzly and Jetty are currently missing/unknown to me.

[siy]

• Spring / Java/JVM

Must admit I'm strongly against this one. Probably worst possible (although very popular) combination. Some framework not so poorly designed and even worse implemented would be much better choice, for example Vert.x.

[bhauer]

@siy Thank you for your input!

For the purposes of hardware performance rating, we feel the popularity of a framework is an interesting attribute. The more popular a framework is, the more likely it has been and will be used in real-world applications that may be running on similar hardware.

That said, not every language that we have represented is highly popular, and it's not crucial that we only select popular frameworks. Put briefly, all else being equal, a popular framework seems a better fit for this use-case than a less popular framework.

I'd like to target having somewhere between 15 and 25 frameworks in the TPR list so that the total execution time of a hardware performance test is "a few hours." With that in mind, I think we have room for multiple frameworks on very popular platforms like the JVM. So adding Vert.x alongside Spring seems reasonable to me.