accumulo icon indicating copy to clipboard operation
accumulo copied to clipboard

Collapse Compaction coordinator code into the Manager

Open ddanielr opened this issue 1 year ago • 4 comments

In elasticity, the coordinator has been collapsed into the manager to reduce complexity around running number of different accumulo servers.

As we have standardized on using external compactions and deprecated internal compactions in 3.1, we should also migrate the coordinator code into the manager.

This should also replace the top level compaction entry from the cluster.yaml with compactor

compactor:
    - q1:
        - localhost1
        - localhost2
    - q2:
        - localhost3
        - localhost4

ddanielr avatar Feb 02 '24 19:02 ddanielr

This is a nice change because it makes the 3.1 release have the following

  • removes the experimental status of external compaction
  • external compactions in 3.1 and 4.0 have a similar user experience

This avoids encouraging use of external compactions in 3.1 and changing the user experience for that feature in 4.0

keith-turner avatar Feb 02 '24 19:02 keith-turner

This is a nice change because it makes the 3.1 release have the following

  • removes the experimental status of external compaction
  • external compactions in 3.1 and 4.0 have a similar user experience

This avoids encouraging use of external compactions in 3.1 and changing the user experience for that feature in 4.0

To expand on the user experience part:

In 2.1 we expect the user to configure a new server type of "compaction-coordinator" to use external compactions. This requires a modification to the cluster.yaml but is considered an experimental configuration change.

In 3.1 we change the feature to be supported and there are no direct user changes.

In elasticity we drastically change the cluster.yaml format and remove the concept of a separate compaction-coordinator server.

By moving the compaction-coordinator change down into 3.1, we reduce a lot of the configuration churn for users and make it part of the expected step of a feature moving from experimental status to supported.

ddanielr avatar Feb 02 '24 19:02 ddanielr

While working on #4225 I found that MiniAccumulo has a compaction coordinator server type in its public API. That will need to be deprecated and its use made to be a noop.

keith-turner avatar Feb 03 '24 22:02 keith-turner

In elasticity, I moved the Coordinator into the Manager in PR #3216. Someone could use this as a guide to account for all of the things that need to be changed.

dlmarion avatar Apr 25 '24 12:04 dlmarion