Will Jones
Will Jones
> Should I use this repo as a production tool or should I leave it and rollback to databricks? We've implemented support for S3 and other object stores. I'm still...
I'm thinking now a good transition path is to: * [ ] Rewrite the delta-rs object stores to just implement the ObjectStore trait from `object-store-rs`, completely removing our own StorageBackend...
Now that we use objectstore, we should implement this upstream: https://github.com/apache/arrow-rs/issues/2615
Resolved by #669.
> Is the functionality of "table creation" still a WIP? No that part should work now. Could you create a new issue for the error you are showing? Make sure...
We can now pass ObjectStores into PyArrow writer. But we need to be able to provide PyArrow / fsspec filesystems into `DeltaTable`. ```python fs = S3Filesystem() dt = DeltaTable("s3://path/to/table", filesystem=fs)...
Resolved by #669
Now that we use objectstore, we should implement this upstream: https://github.com/apache/arrow-rs/issues/2615
Resolved by #607.
Hmm I just learned about an option in PyArrow `use_compliant_nested_type`, which might change some things. I'll look into this soon. Docs: https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetWriter.html#pyarrow.parquet.ParquetWriter Background: https://issues.apache.org/jira/browse/ARROW-11497