fsharp icon indicating copy to clipboard operation
fsharp copied to clipboard

Alphametics tests to handle timeouts for longer tests

Open martinfreedman opened this issue 3 years ago • 2 comments

Changed alphameticstests.fs to make all tests async and added xunit Timeout attribute to longest 3 tests with 2 secs for Puzzle with eight letters and 5 seconds each for Puzzle with ten letters and 199 addends and Puzzle with ten letters exercises.

And switch back on testrunner for this exercise.

martinfreedman avatar Nov 08 '21 14:11 martinfreedman

Which repo and file is it?

This one: https://github.com/exercism/fsharp-test-runner/blob/main/src/Exercism.TestRunner.FSharp/Rewrite.fs#L20

I think timeouts should be discussed at the practise exercise specification, hence my other PR to update canonical json with a timeout property. This might include relative timings between tests as implied here between 8 versus 10 letter tests. The actual implementation would depend on the track maintainers factoring in the built in online test runner timeouts and language performance etc. I would probably default to 10 seconds per test, a propos of anything more specific.

I'm not entirely sure about the timeouts being in the prob-specs definition, but I'll post my thoughts there.

Oh and I completely forgot to mention that the alphametics exercise's test file is automatically generated by a test generator: https://github.com/exercism/fsharp/blob/main/generators/Generators.fs#L61 That means that any changes that are not made to the generator would be overwritten when we re-generate the tests. I think you can ignore that for now, as the F# test generator hasn't yet been updated to use the tests.toml file (I started working on that), but I did want to mention it.

ErikSchierboom avatar Nov 09 '21 10:11 ErikSchierboom

Speaking of the tests.toml file, that would be a great place to configure these timeouts.

ErikSchierboom avatar Nov 09 '21 10:11 ErikSchierboom