Buffer improvements

[binarylogic]

As part of ensuring Vector is enterprise-ready, we need to improve how Vector buffers data. This is core to achieving the level of reliability and durability that enterprises require.

Our disk buffers need a lot of improvement. The initial spike of this work was based on leveldb which has proven problematic for this use case.

This list of items is the high-level work stream related to designing, building, and doing a beta release our buffer improvements for 2022Q1:

• [x] #9477
• [x] #9916
• [x] #9917
• [x] #9918
• [x] #9920
• [x] #9921
• [x] #9922
• [ ] #10324
• [x] #10428
• [x] #10429
• [x] #10430
• [x] #10436
• [x] #10835
• [x] #10895
• [x] #10898
• [x] #10922
• [x] #10939
• [x] #11151

[tobz]

Part of eventually releasing all of this buffer work in full is building confidence in the new disk buffer implementation, as well as deprecating older variants, switching to new ones, and so on.

Roughly speaking, here's the gameplan around the release cadence / feature status:

• 0.20 release (02/08/2022):
  • disk v2 reaches beta status (confident in basic behavior, but not yet fully confident that all logic is resilient to corner cases)
  • switch to in memory v2 as impl for memory buffer type
• 0.21 release (03/22/2022):
  • disk v2 reaches stable status (confident in basic behavior, confident in handling corner cases)
  • feature added to transparently migrate any existing disk v1 buffer to a disk v2 buffer
  • switch to disk v2 as impl for disk buffer type
  • deprecate disk v1 + in memory v1
• 0.22 release (05/03/2022):
  • remove disk v1 and in memory v1 entirely
  • remove disk v1->v2 migration code

The biggest hurdle to the 0.20 release is finishing most of the testing-related issues (#10895, #10898, #10922) and the schema evolution issue (#10939) so that we can have more confidence, and hopefully avoid needing to change the on-disk format, once we move disk v2 to beta status.

The biggest hurdle for the 0.21 release would be encouraging users to experiment with the new disk v2 buffers, as well continuing to recreate real-life workloads to stress test it on our own. Additionally, and perhaps tied as the biggest hurdle, would be writing the migration logic to move disk v1 buffers to disk v2 buffers, and doing so correctly and reliably.

[jszwedko]

Closing this since the epic was complete.