course_os icon indicating copy to clipboard operation
course_os copied to clipboard

Tests fail depending on their order

Open fayalalebrun opened this issue 4 years ago • 3 comments

On the latest revision of the code base, tests will fail around 10-20% of the time, depending on their order. It's hard to trace the reason why, since each time the tests are run, their order is different.

As an idea, it would be useful to be able to choose the order of the tests. And for this order to be printed whenever it's randomly generated.

fayalalebrun avatar Feb 11 '21 11:02 fayalalebrun

The tests are randomized here, if you want to test a specific order you can make your own test.c file. Using seeded random, allowing to give the seed and printing the seed could be a nicer solution than what currently exists. Though feel free to implement what you deem useful.

It's worth noting that the random order was a conscious choice exactly to find more problems like this.

NULLx76 avatar Feb 11 '21 11:02 NULLx76

Indeed, that does sound like the best solution. I'll get started on working on adding an option to seed the generation as well as printing the seed.

fayalalebrun avatar Feb 11 '21 12:02 fayalalebrun

We made tests randomized precisely because we wanted to find such bugs that only happen in specific situations of memory usage. Most likely cause is something in the memory managers. Luckily it's quite easy to attach the debugger to the kernel

jdonszelmann avatar Feb 11 '21 20:02 jdonszelmann