Will Jones
Will Jones
## Bug ### Describe the problem I'm trying to understand the column invariant enforcement in delta lake, so it can implement it in delta-rs. However, I am unable to get...
# Description These change move (most) of the schema code down into Rust, removing (most of) the JSON serialization and making the Python bindings closer to the Rust implementation. It...
# Description I don't believe we actually test this; we only test with `dry_run = true`. **Use Case** **Related Issue(s)**
# Description This crate maintains its own notion of `StorageBackend`, but it might be more than we need to maintain for Delta Lake support. [influxdata/object_store_rs](https://github.com/influxdata/object_store_rs/) implements an `ObjectStore` trait that...
# Description PyArrow Filesystems don't expect paths to be provided as URIs with a scheme prefix (e.g. `s3://`). The only reason that `DeltaTable.to_pyarrow_dataset()` works (if you don't pass a filesystem)...
# Description We currently don't allow external access to the underlying storage of a `DeltaTable`. But for the Python bindings we will need access to be able to pass the...
# Description #587 allows providing a name for a Delta table. However, it won't immediately work for users since we don't register it. We should provide a way to automatically...
# Description Many Pandas types aren't automatically converted into valid Delta Lake types when converted into Arrow tables. For example, Pandas Timestamps are converted into timestamps with nanosecond precision by...
# Description The PyArrow filesystem bindings are only partially implemented: https://github.com/delta-io/delta-rs/blob/main/python/deltalake/fs.py **Use Case** We will need the rest of the methods implemented for the PyArrow writer (`write_deltalake()`) to work. **Related...
# Description For both the datafusion and pyarrow-based writers to support writer protocol v2, we'll need to support enforcing invariants. It seems like the following signature could be reused by...