ArcticDB icon indicating copy to clipboard operation
ArcticDB copied to clipboard

Persistent storage tests: Run against multiple baselines, normalize, and extend test coverage

Open poodlewars opened this issue 10 months ago • 1 comments

PR #1462 made a start on extending persistent storages test coverage.

But the existing persistent storage tests run on a large matrix, checking interop between every 2 elements in this Cartesian product:

{reader, writer}
X
{windows, linux}
X
{python3.6..3.11}
X
{master ArcticDB, single baseline ArcticDB}

They do not support testing where one version writes, and then another version tries to modify that data.

To support tests like that, eg the tests in PR #1462 , we should:

  • Make the tests in the Cartesian product as simple as possible (basic read + write)
  • Introduce a new sort of persistent testing that runs just on Linux with Python3.11
  • On this new setup we can write more involved tests that can freely mutate data (as there won't be another test run also trying to mutate it)
  • We can introduce more testing without massively slow test runs
  • We can run against an arbitrary number of baselines (eg 1.6.x, 3.0.0, 4.0.x)

poodlewars avatar Mar 28 '24 10:03 poodlewars

We will prioritise this after we have a dedicated test engineer

poodlewars avatar Apr 04 '24 12:04 poodlewars