lance icon indicating copy to clipboard operation
lance copied to clipboard

refactor: use list encoder / scheduler directly instead of replicating behavior in map encoder / scheduler

Open westonpace opened this issue 3 months ago • 5 comments

This is a minor simplification of the recently added map support. It does not add any functionality, only simplifies the code slightly. The current implementation creates a StructuralMapEncoder which mimics the StructuralListEncoder. This change instead wraps the StructuralListEncoder (composition instead of duplication). As a result we can get rid of the StructuralMapScheduler. The decoder then simply casts from a list array to a map array.

Since the list encoder is so simple this isn't really saving us a ton of work. However, if we choose to experiment or add new complicated features to the list encoder in the future then this will help us avoid doing the work twice or missing the changes in the map encoder.

westonpace avatar Dec 17 '25 13:12 westonpace

ACTION NEEDED Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

github-actions[bot] avatar Dec 17 '25 13:12 github-actions[bot]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. Credits must be used to enable repository wide code reviews.

CC @xloya do you mind reviewing? I don't feel super strongly about this so if you prefer the old approach that is ok too. Still, I think if we can avoid the map encoder having its own logic that will be helpful.

westonpace avatar Dec 17 '25 13:12 westonpace

Codecov Report

:x: Patch coverage is 84.48276% with 9 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance-encoding/src/encodings/logical/map.rs 84.44% 5 Missing and 2 partials :warning:
rust/lance-encoding/src/decoder.rs 85.71% 1 Missing :warning:
...ust/lance-encoding/src/encodings/logical/struct.rs 83.33% 0 Missing and 1 partial :warning:

:loudspeaker: Thoughts on this report? Let us know!

codecov[bot] avatar Dec 17 '25 14:12 codecov[bot]

I think this optimization is great! There's only one issue regarding the list field naming.

xloya avatar Dec 18 '25 02:12 xloya