soda-core
soda-core copied to clipboard
Decide next steps for Soda Cloud integration tests
Cloud integration tests are run in CI with WESTMALLE: BETTER_THAN_LA_TRAPPE
env var, which should trigger mock soda cloud creation and usage. There is no log/test output that would indicate that this actually happens, these tests just pass the same way as any other tests with no intention of testing the cloud integration. What should be done:
- Make sure it is actually testing comprehensively what needs to be tested (=the cloud integration)
- Ensure clear output of the integration tests to indicate what is being tested.
We need to analyze further and agree on what we do now vs what we postpone for later. Options are
a) Do nothing: the current CI will run the test suite on postgres with WESTMALLE env var enabled. Which means it will run activate Soda Cloud mock for all tests. This adds a bit of coverage. @m1n0 mentions this does not show in the logs. We could add a logging to demonstrate in the test suite execution logs that soda cloud mock indeed is active. But I don't think this logging adds extra coverage.
b) Add test(s) that verifies Soda Cloud interactions using a mock. @vijay what is the status of Soda Cloud testing atm?
c) Integration tests that runs Soda Core & Soda Cloud: probably heavy to set up. Might be included in as part of the Soda Cloud integration tests.
Just a note to a) it indeed does not add extra coverage, but currently to verify whether we actually do what we want to in the test one has to go into code, understand it and debug it. Adding logs about what the mock integration test does surfaces this to anyone examining test logs, making the test less "mysterious"