feat: break stream by max bytes param

[SaintBacchus]

try to fix https://github.com/lancedb/lance/issues/3393

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 92.30769% with 2 lines in your changes missing coverage. Please review.

Project coverage is 78.46%. Comparing base (356d132) to head (0ef5b56). Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/lance-datafusion/src/chunker.rs	92.30%	2 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3435      +/-   ##
==========================================
- Coverage   78.46%   78.46%   -0.01%     
==========================================
  Files         252      252              
  Lines       93917    94032     +115     
  Branches    93917    94032     +115     
==========================================
+ Hits        73696    73778      +82     
- Misses      17226    17262      +36     
+ Partials     2995     2992       -3     

Flag	Coverage Δ
unittests	78.46% <92.30%> (-0.01%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[SaintBacchus]

batch.get_array_memory_size() will return all the batch memory sizes, regardless of whether the batch was sliced.

If we use the average size to estimate the memory usage, it's not accurate.

[wjones127]

batch.get_array_memory_size() will return all the batch memory sizes, regardless of whether the batch was sliced.

If we use the average size to estimate the memory usage, it's not accurate.

Did you see this https://docs.rs/arrow/latest/arrow/array/struct.ArrayData.html#method.get_slice_memory_size ?

[SaintBacchus]

I had missed this method. I will test it later.