lance icon indicating copy to clipboard operation
lance copied to clipboard

Published 20 hours ago • verified publisher icon lancedb

feat: add storage_options to _BaseLanceDatasink, LanceDatasink, LanceCommitter

Open BitPhinix opened this issue 1 year ago • 1 comments

Adds the ability to pass storage_options to _BaseLanceDatasink to allow specifying them for LanceDatasink and LanceCommitter (LanceFragmentWriter already supports passing these).

We need to customize them because some of our datasets are hosted on R2

BitPhinix avatar Jul 19 '24 05:07 BitPhinix

ACTION NEEDED Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

github-actions[bot] avatar Jul 19 '24 05:07 github-actions[bot]

Fixes https://github.com/lancedb/lance/issues/2615 as well

BitPhinix avatar Jul 19 '24 07:07 BitPhinix

Note: the CI failure is unrelated and will be fixed once this is #2620 merged. You should probably rebase on that is merged if you add the integration tests.

wjones127 avatar Jul 19 '24 18:07 wjones127

Added some untested tests, not sure how to run integration tests locally to see if they work

BitPhinix avatar Jul 19 '24 23:07 BitPhinix

Added some untested tests, not sure how to run integration tests locally to see if they work

Thanks. That's described in our development guide here:

https://github.com/lancedb/lance/blob/main/python/DEVELOPMENT.md#running-s3-integration-tests

wjones127 avatar Jul 22 '24 17:07 wjones127