python-tuf icon indicating copy to clipboard operation
python-tuf copied to clipboard

tests: test infrastructure (setup/teardown) improvements

Open jku opened this issue 4 years ago • 1 comments

(Documenting test improvement ideas from #1346)

A lot (most?) of our tests require a test server (or several). This typically means moving metadata from sources to a temp directory, serving the temp directory and finally removing the temporary directories. Modified_Testcase handles parts of that but it seems obvious that A) it does not provide the required functionality (as tests re-implement a lot of it) and B) that it is hard to use correctly (as #1346 and many previous issues prove). The end result is that there are temp directories created inside tempdirectories and then those directories may or may not get removed (often in a fairly random order)

The common needs are these I think:

  • Test class must be able to serve a repository that all its tests run against
  • test instance must be able to serve it's own repository (because it wants to modify the contents)
  • the served repository paths must be exposed so the content can be modified during test

We could have a better Modified_Testcase that just always runs one test server per class, and provides helpers to "intitialize a new repo" (copy files from sources into a directory inside the served directory). This way both server startup/shutdown and tempdir management would be handled in one place: the actual test classes would only have to remember to chain up Setup*() and *Teardown() -- if they even need those when Modified_Testcase is better.

jku avatar Apr 12 '21 14:04 jku

Modified_Testcase was removed in #1790 (eb67b09cf8348516f9fa9137a64232056f092a6d) and tmp dir setup/teardown was simplified in #1792.

For TUF's sister project in-toto (which has similar needs in this regard) I created TestCase mixins with classmethods that aggreate common test setup/teardown functionality. Maybe something to explore here as well?

lukpueh avatar Mar 16 '22 12:03 lukpueh