ice icon indicating copy to clipboard operation
ice copied to clipboard

Skipping slow tests

Open alexmojaki opened this issue 2 years ago • 1 comments

When I run ./scripts/run-tests.sh, all tests show PASSED except two:

tests/test_metrics.py::test_nubia FAILED
tests/test_metrics.py::test_gold_paragraphs SKIPPED (Parses all PDFs - very slow)
  1. test_nubia fails with an obscure pydantic validation error, I had to add a print to understand the problem. How about adding @mark.skipif(not settings.OUGHT_INFERENCE_API_KEY, reason='...')?
  2. Both tests have @mark.slow but this doesn't do anything by default. I assume this is why test_gold_paragraphs has a hard @mark.skip. This doesn't seem great - what do you do when you actually want to run the test? Should a way to skip slow tests by default (with or without using @mark.slow) be investigated?
  3. -m 'not slow' correctly deselects the slow tests, but only if I directly run docker compose and pytest. ./scripts/run-tests.sh -m 'not slow' doesn't work, and the shell (tested both bash and zsh) shows it running with -m 'not\' slow. Looks like this isn't working:

https://github.com/oughtinc/ice/blob/7eab90d22b3bd8ca68f944213e6f9425ff188b3a/scripts/run-tests.sh#L8-L9

alexmojaki avatar Sep 22 '22 14:09 alexmojaki

This definitely seems worth fixing as part of an overhaul of all the scripts: https://github.com/oughtinc/ice/issues/7#issuecomment-1256665522

lslunis avatar Sep 23 '22 20:09 lslunis