Changes / Updates / Dates for Round 19

[NateBrady23]

I'll be editing this pinned issue to link to other issues or pull requests that may affect tests for this round.

Round 19 testing will begin on or around May 1st. Last call for changes to be reviewed and merged is April 29th.

• #4984 - Updated major versions of MySQL and MongoDB
• #5015 - The requirements are now just on GitHub, and the results web site links there instead of having a duplicate copy.
• #5167 - Single and Multi Query tests should be reading the entire row from the database. This was stated in the rules but have added an extra sentence to make it clearer that selecting just the randomNumber is incorrect.
• #5199 - Postgres to version 12
• Citrine environment from CentOS to Ubuntu 18.04
• #5285 - Wrk improvements and MongoDB update
• Azure environment from Ubuntu 16.04 to 18.04 and changing the terraform script from using latest image to a specific version for better reproduction of test results. Apr 23, 2020

Blockers for Round 19

• [x] Investigating issue related to #4092 and other start/stop issues
• [x] Add a message to tfb-status with some advance notice before PR submissions are closed for the round

Round 19 results have been posted: https://www.techempower.com/benchmarks/#section=data-r19

[bhauer]

• #5015 The requirements are not just on GitHub, and the results web site links there instead of having a duplicate copy.

[zloster]

Hi all. I want to bring attention to the following issues:

1. There is no response for quite long time on these two: https://github.com/TechEmpower/FrameworkBenchmarks/issues/4092#issuecomment-484549733 and https://github.com/TechEmpower/FrameworkBenchmarks/issues/4885#issuecomment-506957575 I'm not able to pinpoint the problem that is causing the described behaviour.
2. https://github.com/TechEmpower/FrameworkBenchmarks/issues/4083#issuecomment-549923026 I would like to hear how to proceed with this. It concerns the frameworks using Java and servlet container. The data suggests there is a difference.

[volyrique]

Are there any plans for OS upgrades? Newer kernel versions might be affected differently by the security countermeasures that led to a lot of performance regressions before. CentOS 8 got released more than a month ago (CentOS 7 is used in the physical hardware environment), and I believe that we have more recent options for the Azure VMs (which use Ubuntu 16.04 if I am not mistaken).

[bhauer]

@volyrique Indeed, we are presently considering switching over to Ubuntu 18.04.

[hushino]

Update package.json of Hapi from "hapi": "16.1.1" to "@hapi/hapi": "^18.4.0" package.json of hapi tests

[volyrique]

How about any plans to visualize the cached queries test type?

[bhauer]

@volyrique Good question. I'll see if I can get that enabled for the next round.

[OneCricketeer]

Please upgrade Dropwizard to at least 1.3.17 or to 2.0

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Java/dropwizard/pom.xml#L17

[OneCricketeer]

Please upgrade Quarkus to a 1.x.y release

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Java/quarkus/pom.xml#L10

[Liroo]

Adding uWebSocket @alexhultman

[NateBrady23]

Round 19 will run in April. Pull requests not ready by 4/3 won't be guaranteed to be part of Round 19.

[zloster]

@nbrady-techempower @bhauer https://github.com/TechEmpower/FrameworkBenchmarks/issues/4092#issuecomment-551295713 The problem with the TCP/IP connections is still there and there is no comment from your side. I've found a possible reproduction steps. It's just a configuration change. Please, take a closer look.

[NateBrady23]

We're taking Citrine down for maintenance and working on some pressing issues that may have a negative impact on the next round. We'll be postponing round 19 for a couple of weeks, so please feel free to keep submitting pull requests. We'll have messaging on Citrine and in this thread when things are back up.

[sebastienros]

pressing issues that may have a negative impact on the next round

Can you share the details if we want to repro the environment?

[msmith-techempower]

Can you share the details if we want to repro the environment?

We will share any changes we make.

[NateBrady23]

@sebastienros We're working more closely on this issue right now https://github.com/TechEmpower/FrameworkBenchmarks/issues/4092 and anything else we do will be completely transparent.

[OneCricketeer]

I would contribute a PR, but tied up with other priorities. Can someone take care of Dropwizard, please?

[joanhey]

Before close the PRs for the next Round.

Please add a message in advance in https://tfb-status.techempower.com/ Like the one that you place when you're doing maintenance.

[NateBrady23]

Thank you @joanhey for making that request here. I've added it to the top as a reminder to myself.

[NateBrady23]

We're moving forward with Round 19:

Round 19 testing will begin on or around May 1st. Last call for changes to be reviewed and merged is April 29th.

[joanhey]

Update Azure to Ubuntu 18.04 like in Citrine.

[Phara0h]

Are you hard set on the deadline for today? I was planning on having mine by the end of the weekend. Pretty sure I have the fastest web framework on the web right now, minium fastest with node.js.

[NateBrady23]

@Phara0h A continuous run started today on Citrine. The next run will be reviewed for Round 19, so technically you do have a few days, but it would not give you any time to review data and fix problems.

[NateBrady23]

Round 19 results have been posted: Round 19 results have been posted: https://www.techempower.com/benchmarks/#section=data-r19

Unpinning this issue but will leave open for a bit. Pinning a new issue to start tracking changes for Round 20