redpanda icon indicating copy to clipboard operation
redpanda copied to clipboard

storage: Refactor segment index

Open Lazin opened this issue 4 months ago • 9 comments

This PR is a preparation step for the introduction of segment index compression. It moves some logic closer to the data from segment_index to the index_state. It also extractsindex_columns class out of the index_state. The index_columns component contains three vectors (relative_offset_index, relative_time_index, position_index). Previously, the index_state maintained them alongside all other state and the index_state. That makes it impossible to evolve the index because the state is entangled with quite complicated business logic (esp. with timestamp handling). The extraction of index_columns will allow us to evolve this component independently from index_state and all associated complexity.

The next PR in a series will introduce different implementation of the index_columns which will use deltaFOR encoding to save memory.

Backports Required

  • [ ] none - not a bug fix
  • [ ] none - this is a backport
  • [ ] none - issue does not exist in previous branches
  • [ ] none - papercut/not impactful enough to backport
  • [x] v24.2.x
  • [x] v24.1.x
  • [ ] v23.3.x

Release Notes

  • none

Lazin avatar Oct 16 '24 16:10 Lazin