Explore moving all tablet in memory state an synchronization into a single class

[keith-turner]

The Tablet code has a mixture of I/O code and in memory state (like active walogs, active in memory maps, current files, minor compactions in progress, refreshes in progress, etc). Its hard to analyze this synchronization code to ensure things like does any I/O happen while holding a lock (which is not desired).

Moving all of the tablets in memory state+sync into a single class that could be unit tested and analyzed may be one way to improve this.

Doing this work in the elasticity branch will be easier as a lot of functionality has been removed from the tablet server.

[EdColeman]

Should this target an earlier branch? Maybe even a 2.1.x? While it may be easier to implement for elasticity, if this is addressing a source of possible race conditions (so, a bug fix) or reduces the change of introducing errors, having this earlier would benefit users (and help us) until elasticity is adopted. 2.1.x is an LTM release, so we are going to need to support it for quite a while.

This may be more complicated for something targeting 2.1.2, but targeting a 2.1.3 and then merging forward seems that it could provide long-term improvements for this complicated code.

[keith-turner]

While working on trying to determine the best angle to get started with this, one important thing I realized is that a tablet only has the following types of threads that interact with it.

• Scan thread
• Write thread
• Minor compaction thread
• Refresh thread
• Close thread
• Instantiation thread that creates a tablet on a tserver

Prior to elasticity there were more types of threads to worry about (bulk import, compaction, and split). The threads above need to synchronize on different in memory state and this is interleaved with compute+I/O. Started experimenting with this structure. The idea behind this structure is that each of the thread type uses a coordination object to interact with the shared in memory state of the tablet. Maybe this structure would cleanly separate the in memory state+sync from the I/O+compute code, not sure though.

[keith-turner]

Should this target an earlier branch? Maybe even a 2.1.x?

It would be more difficult to make this change in 2.1 because the Tablet has alot more code related to synchronizing compaction, splits, and bulk imports that does not exists in 4.0.0. So I think its good to do the refactoring for 4.0.0 or later. If any existing bugs are noticed during the refactoring, then those could possibly be fixed in 2.1.x.