lance
lance copied to clipboard
feat: add backwards compatibility with 0.13.0 v2 datasets
A breaking change was made to the binary encoding in 0.14.0. v2 was still labeled as experimental at the time. However, a significant number of people have run into this issue and it can be difficult to migrate since it requires recreating the data. This PR adds support for reading the old style.
Note: this will only help reading lance datasets and not lance files. This is because the lance file reader uses the default projection (tries to infer which columns to read from the schema) and a lance dataset uses the field id -> column index mapping in the manifest. The default projection logic has changed and so that approach no longer works.