fix: various improvements for 2.1 take, put FSL back into the compressors

[westonpace]

Previously we moved all FSL handling out into the rep/def levels. Now we're back at it again, but this time we've put the FSL handling back into the compressors. It turns out we really don't want to store rep/def levels for every element in a vector.

There are also some minor perf. improvements to bitpacking and we disable bitpacking if the input is really small (mainly for testing purposes).

Another significant change is that the miniblock encoding now allows for a variable number of buffers instead of requiring things to be zipped into one.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 80.80960% with 256 lines in your changes missing coverage. Please review.

Project coverage is 78.82%. Comparing base (a6101e5) to head (f9e179b).

Files with missing lines	Patch %	Lines
...coding/src/encodings/physical/bitpack_fastlanes.rs	40.82%	129 Missing :warning:
rust/lance-datagen/src/generator.rs	0.00%	55 Missing :warning:
...ust/lance-encoding/src/encodings/physical/value.rs	95.07%	27 Missing and 1 partial :warning:
.../lance-encoding/src/encodings/logical/primitive.rs	95.00%	1 Missing and 16 partials :warning:
rust/lance-encoding/src/format.rs	53.33%	14 Missing :warning:
rust/lance-encoding/src/data.rs	88.67%	6 Missing :warning:
rust/lance-encoding/src/encoder.rs	73.33%	4 Missing :warning:
rust/lance-encoding/src/buffer.rs	96.42%	1 Missing :warning:
rust/lance-encoding/src/decoder.rs	91.66%	1 Missing :warning:
rust/lance-encoding/src/encodings/physical.rs	0.00%	1 Missing :warning:

Additional details and impacted files

@@           Coverage Diff            @@
##             main    #3450    +/-   ##
========================================
  Coverage   78.81%   78.82%            
========================================
  Files         251      251            
  Lines       92821    93752   +931     
  Branches    92821    93752   +931     
========================================
+ Hits        73159    73900   +741     
- Misses      16686    16869   +183     
- Partials     2976     2983     +7     

Flag	Coverage Δ
unittests	78.82% <80.80%> (+<0.01%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[westonpace]

Superceded by https://github.com/lancedb/lance/pull/3488