soda-core icon indicating copy to clipboard operation
soda-core copied to clipboard

Decide next steps for Soda Cloud integration tests

Open m1n0 opened this issue 2 years ago • 2 comments

Cloud integration tests are run in CI with WESTMALLE: BETTER_THAN_LA_TRAPPE env var, which should trigger mock soda cloud creation and usage. There is no log/test output that would indicate that this actually happens, these tests just pass the same way as any other tests with no intention of testing the cloud integration. What should be done:

  • Make sure it is actually testing comprehensively what needs to be tested (=the cloud integration)
  • Ensure clear output of the integration tests to indicate what is being tested.

m1n0 avatar Mar 28 '22 18:03 m1n0

We need to analyze further and agree on what we do now vs what we postpone for later. Options are

a) Do nothing: the current CI will run the test suite on postgres with WESTMALLE env var enabled. Which means it will run activate Soda Cloud mock for all tests. This adds a bit of coverage. @m1n0 mentions this does not show in the logs. We could add a logging to demonstrate in the test suite execution logs that soda cloud mock indeed is active. But I don't think this logging adds extra coverage.

b) Add test(s) that verifies Soda Cloud interactions using a mock. @vijay what is the status of Soda Cloud testing atm?

c) Integration tests that runs Soda Core & Soda Cloud: probably heavy to set up. Might be included in as part of the Soda Cloud integration tests.

tombaeyens avatar Mar 29 '22 10:03 tombaeyens

Just a note to a) it indeed does not add extra coverage, but currently to verify whether we actually do what we want to in the test one has to go into code, understand it and debug it. Adding logs about what the mock integration test does surfaces this to anyone examining test logs, making the test less "mysterious"

m1n0 avatar Mar 29 '22 11:03 m1n0