Allow a server to host segments on multi storage tiers

[klsince]

As of today, a table with multi-tiers has to use different server pools to support multi storage tiers, and relies on SegmentRelocator to move the segments among those server pools to put them on the right storage tier. An example cluster setup may be like using a pool of servers with SSD as a tier and another pool of servers with HDD as a second tier.

But this model doesn't work when a server have both SSD and HDD, and wants to define multi tiers backed by them.

We'd like to address this issue. And one possible way is to implement a SegmentDirectoryLoader that's aware of multi data dirs; and enhance current way of tier migration to move segments across tiers even if they are not relocated to a different server, i.e. to be done on the same server locally.

[klsince]

found another related issue: https://github.com/apache/pinot/issues/6343

[somandal]

hi, this issue had a design doc earlier, but now i see it's been removed and I don't seem to have permissions to access it anymore. Can you provide access?

[klsince]

Proposed some ideas about how to tackle this issue in design doc. Feel free to comment.

[klsince]

The major functionality is completed. What's remaining is about how to cleanup orphan segments, which is a related problem but orthogonal, because orphan segments can happen in many situations other than tier migration, as detailed in the design doc above. I'll address that separately.