feat: Refactor the read/write lance API based on the Lance Rust SDK

[plotor]

Changes Made

Related Issues

#4841

Checklist

• [ ] Documented in API Docs (if applicable)
• [ ] Documented in User Guide (if applicable)
• [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation
• [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review)

[Jay-ju]

It just sounds wonderful

[codecov[bot]]

Codecov Report

:x: Patch coverage is 6.66667% with 14 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 74.23%. Comparing base (43b251b) to head (4f3cfa0).

Files with missing lines	Patch %	Lines
daft/dataframe/dataframe.py	6.66%	14 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4842      +/-   ##
==========================================
- Coverage   74.27%   74.23%   -0.04%     
==========================================
  Files         959      959              
  Lines      123429   123405      -24     
==========================================
- Hits        91673    91613      -60     
- Misses      31756    31792      +36     

Files with missing lines	Coverage Δ
daft/dataframe/dataframe.py	77.92% <6.66%> (-0.95%)	:arrow_down:

... and 13 files with indirect coverage changes

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[malcolmgreaves]

@plotor @Jay-ju this is marked as a draft so we haven't reviewed yet. Are y'all still working on this or is it in a state that's ready for review?

[plotor]

@plotor @Jay-ju this is marked as a draft so we haven't reviewed yet. Are y'all still working on this or is it in a state that's ready for review?

@malcolmgreaves I'm currently working on this, and I'll update this PR (or submit a new one) once it's ready.

[madvart]

Hi @plotor - Checking to see how this is going?

[plotor]

Hi @plotor - Checking to see how this is going?

The refactoring of read_lance based on the Rust SDK is about 80% complete, and it should be ready for review by the end of this month.