amti icon indicating copy to clipboard operation
amti copied to clipboard

Add automated tests

Open nalourie-ai2 opened this issue 4 years ago • 0 comments

Until now, amti's testing has been entirely manual. There have been three reasons for this decision:

  1. amti began as a tool I built for myself. Originally, I wanted to demo the idea of reproducible, version controlled crowdsourcing pipelines and to make my own crowdsourcing research reproducible. I open sourced amti so that others could run my pipelines, to share the idea of reproducible crowdsourcing, and in case people found it helpful.
  2. amti is still in initial development (has not had a 1.0 release), and manual testing means less effort expended on maintaining automatic tests.
  3. Most good tests for amti require mocking the MTurk APIs or running against the MTurk sandbox; so, good tests are more work to write than in other situations (and thus it's more valuable to keep testing burden low during initial development).

That said, as adoption grows it's more important to ensure amti's reliability. Similarly, amti needs high quality automated tests before any possible 1.0 release.

Since amti will still undergo some major refactoring before 1.0 (see Issue #24 for example), it's worth discussing tests people plan to write here beforehand, to avoid wasting effort.

Here are my thoughts on how to increase test coverage:

  • amti.utils contains lots of small utilities that often don't require mocks and whose APIs are unlikely to change. They can be tested first with Python's unittest module.
  • Other parts of the CLI work only locally (e.g., amti.actions.extraction), don't require mocks, and probably won't change much. They're also good candidates for initial tests.
  • Mocked tests help local development because they're fast and don't require a network connection; however, tests hitting the worker sandbox provide the most assurance that the code works correctly. We should focus on tests against the worker sandbox over mocked tests.
  • Tests against the worker sandbox should be run infrequently (i.e. after committing or when merging a PR) and thus need to be separated from local unit tests used for quick, frequent feedback during development.

nalourie-ai2 avatar Apr 28 '20 01:04 nalourie-ai2