cvra
Compatibility tests between python and C implementations
We should also test the implementations with each other by additional unit tests.
To repeat what I said:
I would do it by making a C to Python binding to the target code and then write all compatibility tests in python.
That said, I am not sure if it is worth it. It is more into the realm of integration or end to end testing, which have a much lower ROI than unit tests. Since this project is already running late I wonder if it is very wise to do it...
I am not sure if it is worth it.
I think it is still better to use python generated data instead of assembling all test data by hand. We don't have to do it for both ways. Especially the communication from client to target is more complex and should be tested by unit tests. The test data can be automatically generated and stored as a C include file, containing the raw data and the expected values.
I would do it by making a C to Python binding to the target code and then write all compatibility tests in python.
Though, this sounds like much work.
Maybe for those tests it is worth it: https://github.com/cvra/can-bootloader/blob/master/tests/integration_tests.cpp
For other tests I still feel like writing tests in CPP is better, because they are generally simpler and do not contain too much data. This one for example: https://github.com/cvra/can-bootloader/blob/master/tests/datagrams_command_tests.cpp
Sure. I don't mean to replace the existing tests, just add at least some tests for python and C interoperability to avoid problems like 97670a40a5904461167364b025c870735d3891e9, where the C tests couldn't detect the type problem. I'm ok with manually generated data, though it would be nice, if it could be done automatically on each test run.
So my preferred approach would be to use ctypes (https://docs.python.org/3.4/library/ctypes.html) and generate a shared library, which would then be called by the python tests. What do you think ?
@antoinealb ctypes is pretty simple to use. we use it in our beacons library for example, as long as the wrapped api isn't too big or complicated this is probably the best way to go. i could lend a hand if needed (=
Although I have to admit I would prefer to move on to further steps in the planning... This is already lagging behind the schedule.
Sure, now it's not too important anymore, since we already have hardware to test with. Though for future projects we should do it this way from the beginning.