webonnx
Add testing framework to compare against known-good implementations
Is your feature request related to a problem? Please describe.
In order to be able to test correctness of the implementation it would be a good idea to be able to automatically compare it to some other known-good reference point (ideally the ONNX test suites but less ideally some other mature implementation, i.e. https://github.com/sonos/tract for instance).
Describe the solution you'd like
This could simply be a test that runs a set of ONNX models with specific inputs and outputs in WONNX and some other runtime, and then compares the result. I have tested this approach already here: https://github.com/pixelspark/nnx/blob/main/src/main.rs#L126 (and here's how to do inference with tract).
Describe alternatives you've considered
Well, writing tests that check every corner case by reading the spec very carefully :-)
@haixuanTao One other interesting thing is that I believe tract can also do the inference of dimensions (we require onnx-simplifier), might want to have a look at how they are doing it!
There is also an 'official' ONNX test harness, which apparently you can plug your own backend in to test it: https://github.com/onnx/onnx/blob/master/docs/ImplementingAnOnnxBackend.md. Probably requires some Python glue code.
Ok, so, I think I'm going to plug to the backend tester with python and put it in the CI :)
Just leaving a message here, that I'm working on implementing the onnx backend test suite. But I'm in the middle of a professional move at the moment that is taking a bit of time 😅
I'm going to have more time in February 🙂