Need a way to test UNKNOWN result for conditional mutations

[keith-turner]

Conditional mutations are used for metadata updates now. Conditional mutations can return a result of UNKNOWN. The Ample code that handles conditional updates has a method that accepts a lambda for handling these unknown conditions. This code shows an example lambda.

These lambdas in the code currently have no test. Need to devise a way to test these and then start writing test for all of the conditions. It is possible that Ample may need to change to make this condition more testable.

[cshannon]

@keith-turner and I discussed this today and came up with a plan to create a prototype test Ample that can be used to run tests against a different table that is similar to how the latest Fate changes added tests to run against a different Fate instance and table than the system tablet (#4049, #4202, etc)

The goal would be to mostly use the real Ample code but to add in some points (maybe some protected methods) to allow customizing behavior and testing real code but against a different table than actual metadata. Using the real metadata table is problematic since when the mini cluster is running it writes real data and interferes with the tests and what is expected. Using a test table allows us to crate/drop and do whatever we need to test algorithms without interfering.

We are thinking something like:

Ample ample = TestAmple.create(Map.of(DataLevel.USER, "mytable"))


// use ample to setup expected data prior to test

ample.mutateTablets()...


// call metadata algorithm lke split w/ ample

split(ample, splitData)


// verify after state using test ample

ample.readTablets()

I will start working on a prototype and if I get something usable can open up a draft PR.

[keith-turner]

@keith-turner and I discussed this today and came up with a plan to create a prototype test Ample that can be used to run tests against a different table that is similar to how the latest Fate changes added tests to run against a different Fate instance and table than the system tablet (https://github.com/apache/accumulo/pull/4049, https://github.com/apache/accumulo/pull/4202, etc)

This solves a more general problem than what was described initially in the issue. We think this more general solution can solve the initial problem that was described and more. The elasicity branch stores much more state in the metadata table and being able to more comprehensively test the algorithms that modify this state stored in the metadata table will be huge improvement for verifying the correctness of the elasticity changes.

Hoping the proposed design can improve existing tests like #4321, #4323, and #4352 and also make it possible to test the metadata algorithms for compaction, bulk import, split, merge, tablet location update, minor compaction, etc more comprehensively. Without something like this it would be hard to get full coverage for edge case for all of these metadata algorithms.