lance icon indicating copy to clipboard operation
lance copied to clipboard

feat: add backwards compatibility with 0.13.0 v2 datasets

Open westonpace opened this issue 1 year ago • 0 comments

A breaking change was made to the binary encoding in 0.14.0. v2 was still labeled as experimental at the time. However, a significant number of people have run into this issue and it can be difficult to migrate since it requires recreating the data. This PR adds support for reading the old style.

Note: this will only help reading lance datasets and not lance files. This is because the lance file reader uses the default projection (tries to infer which columns to read from the schema) and a lance dataset uses the field id -> column index mapping in the manifest. The default projection logic has changed and so that approach no longer works.

westonpace avatar Aug 19 '24 14:08 westonpace