[CX_HARDENING] - Document and Audit Testing Frameworks

[jparr721]

What is this task and why do we need to work on it?

The various testing apparatus that are used in the codebase are opaque, and very few team members know fully how they work. Furthermore, these tests, such as memory_network, lack clear documentation explaining what is going on, and what is actually being tested, leading to complexities when these tests fail while making changes. We need to document and audit the code that is running our integration tests because we cannot be sure yet that they are as correct as possible.

Consider as an example the recent testing instability. Part of that was due to a bug in the testing framework, which was fixed in this PR. The reliance on knowledge gained over time is useful when considering logical changes. But our testing needs to be robust, well-documented, and thorough such that a new team member can contribute without concern of introducing instability into the codebase. Right now, only seasoned team members would have likely been able to fix this issue due to a lack of overall knowledge of the tests. As a result, it is imperative that we get these systems documented, ideally as a post-hoc design document that outlines the why. This will not only give new-hires and current employees a much clearer understanding of how these tests work, but also empower the team to suggest improvements, and devise better tests of our core systems.

What work will need to be done to complete this task?

We need a notion design document (found in the consensus team page) that outlines in detail the exact inner workings of the integration and logical testing frameworks. We should provide a patient written record of how these systems work and their known shortcomings. Ideally, we should definitely get some tickets out of this for improvements to the tests that can be made as well.

Are there any other details to include?

No response

What are the acceptance criteria to close this issue?

One or two notion design documents (following the template) which give a thorough deep dive of the testing system, and shared with the team.

Branch work will be merged to (if not the default branch)

No response