cml icon indicating copy to clipboard operation
cml copied to clipboard

Separate unit tests from acceptance tests

Open 0x2b3bfa0 opened this issue 4 years ago • 3 comments

CML is a project that relies heavily on external infrastructure, and some acceptance tests are practically unavoidable if we want to make sure that nothing breaks. Nevertheless, I was wondering if we could mock as much as possible to separate unit from acceptance tests, enabling users to check their contributions for errors without having to care about those services.

0x2b3bfa0 avatar May 16 '21 16:05 0x2b3bfa0

Solving this would probably imply repourposing ~test.yml~ deploy-test.yml from ~#526~ #530 to run unit tests first and relying on GitHub Environments just for acceptance tests.

0x2b3bfa0 avatar May 16 '21 17:05 0x2b3bfa0

Tensorboard tests are flaky because of rate limits: yet another reason to separate unit tests from acceptance tests and only run the latter on demand.

0x2b3bfa0 avatar May 20 '21 09:05 0x2b3bfa0

I don't find a proper way to do such unit tests aside recording the API responses as I have said so many times. If you are willing to do so @0x2b3bfa0 you have to setup sinon.js to mock the real responses.

DavidGOrtega avatar May 21 '21 13:05 DavidGOrtega

Resolved with https://github.com/iterative/cml/pull/1207

0x2b3bfa0 avatar Feb 17 '23 15:02 0x2b3bfa0