[TESTING] Generate test cases from separate lines

[Felienne]

Once we have reorganized the test cases #841, we could consider extracting the list of single line programs that we already test and generate (random?) combinations of lines to test.

This has as benefits:

• We do not have to manually combine lines (saving lines of test code)
• We cover more cases

We would have to disallow some cases (like no echo before print in level 1). Maybe that ends up being very complex but we could see where it takes us?

[jpelay]

So, as I understand, the test only cover one liners, or just one type of case (like test_echo, test _assign, etc) and what is needed is to combine them randomly and see if they work, like:

program = ""
for x number of times:
   grab random test_case
   join program with this test_case
check if program is valid

Can this be done inside the test cases for each level, or a more general that covers all of them?

Thank you :)

[Felienne]

Great questions!

So, as I understand, the test only cover one liners,

Some tests also test longer programs, like this one:

https://github.com/Felienne/hedy/blob/412eaaaf85c1b61dce531a5d964634f7dc5ccf95/tests/tests_level_04.py#L13

But it is all a bit "random" just the things I thought about, not something systematic.

what is needed is to combine them randomly and see if they work, like:

program = ""
for x number of times:
   grab random test_case
   join program with this test_case
check if program is valid

Yes! The joining is interesting though... In the first few levels, all commands are a line so they can easily be mixed, but once we get indentation the joining is not so easy. But let's just start with some easy levels and see what it brings.

Can this be done inside the test cases for each level, or a more general that covers all of them?

I would say within the levels is a good start, if we get too much duplication, we can always move it. If you want to, you could also follow my approach for the multi_level_tester and implement a generator in the base class and call that from subsequent levels.

Let me know if you have enough information like this!

[jpelay]

Oh, thank you that clarifies a lot. And yes, I didn't realize indentetation might be a problem, but I'll try to find a way. I think with this I can start working on something! Thank you for your answers!