typing_copilot icon indicating copy to clipboard operation
typing_copilot copied to clipboard

Add a tracing-based test harness for testing typing_copilot behavior

Open obi1kenobi opened this issue 3 years ago • 0 comments

typing_copilot is already quite a complex tool, and is not trivial to test for correctness. To make it easier to test, split it into a "driver" which knows how to interact with mypy and the filesystem, and essentially a "library" that is fed data and is expected to act on it appropriately.

Then, different kinds of drivers could exist:

  • a driver that actually does read/write to disk and invoke mypy
  • a driver that wraps the one described above, but also records the actions taken and their outcomes into a trace file
  • a driver that reads a trace file and asserts that the library follows the trace as expected

The second driver could then be used to automatically generate traces of bug reports. The generated trace files could then be added to the test suite, and the third kind of driver could be used to ensure ongoing correctness in the face of the observed behaviors.

obi1kenobi avatar Jul 15 '21 18:07 obi1kenobi