perfect-mysql Data updates benchmark result is invalid

[bgrainger]

In the latest benchmark run the fastest MySQL-based "Data updates" entrant is perfect-mysql, by a factor of almost 3× over the next-fastest entrant. Meanwhile, for "Single query", perfect-mysql is in 99th place, which suggests that the "Data updates" result is anomalous.

IMO, perfect-mysql doesn't meet the requirements for "Test type 5: Database updates" and should be brought into compliance or removed from the benchmark.

4. The request handler must retrieve a set of World objects, equal in count to the queries parameter, from the World database table.

I don't see this happening anywhere in this request handler. It calls updateOneFromWorld n times, but that function simply executes one UPDATE statement. There is no SELECT statement that I can see, particularly one that occurs before the DB is updated.

6. At least the randomNumber field must be read from the database result set.

The code on lines 138-143 fakes a returnObj without reading it from the database. The presence of that code strongly suggests that randomNumber is not being read from the database result set. No such code should be necessary in a conformant implementation.

7. Each World object must have its randomNumber field updated to a new random integer between 1 and 10000.
8. Each World object must be persisted to the database with its new randomNumber value.

There does not appear to be any in-memory update of a "World" domain object that happens before the DB is updated. The update is a direct update of a DB row (on line 113) with an interpolated string, not a dynamically-generated query based on ORM-style persistence.

(cross-posted from https://github.com/TechEmpower/FrameworkBenchmarks/pull/4048/files#r251586616)

[NateBrady23]

Thanks for the find @bgrainger

If we can't get to it (other work going on right now) we'll make sure it's not included in the next round.

Leaving this open until it's fixed.

[zloster]

@nbrady-techempower Idea/suggestion for extending the toolset validation. Get and log the basic stats for the number of executed statements. The related info could be found here:

• PostgreSQL https://www.postgresql.org/docs/10/pgstatstatements.html
• MongoDB https://docs.mongodb.com/manual/reference/program/mongostat/#bin.mongostat
• MySQL (Note: I've tried to find more relevant info but I couldn't) https://dev.mysql.com/doc/refman/5.6/en/performance-schema.html This is for 5.6. Starting with 5.7 some of this stuff is accessible in sys schema which is installed by default.