Subbu Subramaniam

@9aman please refer to the comment https://github.com/apache/pinot/issues/10147#issuecomment-2197768243 in this issue. We should not be updating the idealstate before the segment has completed build and is present in the deep store.

> @mcvsubbu > > 1. In ZooKeeper, segment metadata will use **COMMITTING** as the status for segments being built, distinguishing them from **DONE** segments that are already in deep store....

Duplicate of https://github.com/apache/pinot/issues/8806 ? Duplicate of https://github.com/apache/pinot/issues/5647 (as you mention in description)

At a high level, we could treat each topic-partition pair as a single consuming partition in pinot. We will need some more abstractions to be able to support different encoding...