More Random Tests and Comments

[ansvonwa]

• If speed fluctuates, add a comment with a list of results of random tests.
• If it is known that a solution does not always work (or there may be edge cases), add a comment with successful tests vs number of tests, as mentioned in #19 .

Example:

...
-- Size: 42
-- Speed: 204
-- Speed Tests: 203, 204, 204, 205, 203, 203
-- Success Rate: 123/150
...

[soerface]

What is your suggestion regarding the file name? How many speed tests should we demand? How is "fluctuates" defined - every solution that is non-deterministic? Only ones with a certain standard deviation? How high should the deviation be? How is the success rate defined - the game itself just does 25 tests and does not count the number of failures. Do you expect from everyone to run the same script 6 times, every time counting the number of red bars, and then leaving the game, since running it again will result in a failure?

Tough issue…

[ansvonwa]

• What is your suggestion regarding the file name? no changes (size as in the comment -- Size: -- Size: , speed as in -- Speed: )
• How many speed tests should we demand? only one run
• How is "fluctuates" defined - every solution that is non-deterministic? If the program yields to different times. That includes non-deterministic solutions as well as deterministic solutions with times depending on random input values
• Only ones with a certain standard deviation?
• How high should the deviation be? I would not like to set hard rules for that, in my view, it makes more sense to write down how many tests you did and their results, so that everyone else can contribute to that by appending more tests and adding the results
• How is the success rate defined - the game itself just does 25 tests and does not count the number of failures. <number of positive results> / <number of tests> but those values should not be divided, they should both be kept, separated by the slash, so that one can do more tests and easily add up the results
• Do you expect from everyone to run the same script 6 times, every time counting the number of red bars, and then leaving the game, since running it again will result in a failure? No, just, if there are red bars, count them and add a note, the number of green bars down (followed by a "/25" Then, these results may be improved later.

I am sorry I did not point that out clearer before, but the main goal is NOT to force everybody to do complicated tests, but to provide a consistent way to save tests and open the possibility to add more tests later.

For example, in the Success Rate in 26: Budget Brigade 2/size-8_speed-184.asm can not be improved further, because the exact values (how many of how many tests were successful) are not mentioned. I would propose not to write "XX%" but "XX/YY".

The list in 30 - Fill the Floor/size-7_speed-229.asm is good, because it can be expanded. But I would suggest to keep a note -- Speed: with the rounded average of the list (to have one concrete value to use i.e. for plotting and file naming) and if (and only if) there is the need for a list, add a comment -- Speed Tests: with the list (that may contain one or more entries). We also need that second comment to highlight solutions with fluctuating results In the example that would be:

-- Author: landfillbaby
-- Size: 7
-- Speed: 229
-- Speed Tests: 229, 220, 239, 229, 227, 229, 246, 220, 213, 237

(average speed is 228.9, so, rounded 229)

[ansvonwa]

Next problem: Always working solutions may be dominated by sometimes working solutions! As a hot fix, the user is asked if he still wants to add the solution. Does anyone have better proposals? Add a note in the filename if the solution does not always work? Let the script ignore dominating solutions that do not always work?