delta
delta copied to clipboard
delta-io
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
## Description Resolves https://github.com/delta-io/delta/issues/1281 Updating the schema of a field involved in generated columns is not allowed, because it could change the semantics of the generated field. However, the fields...
## Description Resolves #1188 ### PR changes Added a `details()` function to the Python and Scala APIs of `DeltaTable` similar to running `DESCRIBE DETAIL` to get information about a table...
## Description Upgrade the Spark dependency version to 3.3.0. Following are the major changes: * Test fixes to change the expected error message * `VacuumCommand`: Update the parallel delete to...
Hi team, We have a CDC pipeline which is used to update records in AWS S3. Multiple collections are processed in parallel using Python + Spark. Below are the version...
## Description The current implementation of `S3SingleDriverLogStore.listFrom` lists the entire content of the parent directory and filters the result. This can take a long time if the parent directory contains...
## Description Support Auto Compaction described in: https://docs.databricks.com/delta/optimizations/auto-optimize.html#how-auto-compaction-works We can support Auto compaction via a new post commit hook and OptimizeCommand with less size threshold. `spark.databricks.delta.autoCompact.enabled` (default: false) `spark.databricks.delta.autoCompact.maxFileSize` (default:...
Code Cleanup / Refactoring: - Use vals (not vars) - Fix scaladocs - Use constants - Check table availability with txn.deltaLog.tableExists (not txn.readVersion > -1) - Import reorg - Remove...
## Bug I am working on testing new features for deltalake 2.0 and facing below issue while trying to initialized spark session with “configure_spark_with_delta_pip”. Any advice on what might be...
## Description Resolves https://github.com/delta-io/delta/issues/860, (and maybe https://github.com/delta-io/delta/issues/1239 too) Currently a non-nullable field inside a nullable struct is enforced to be not null. However, the way Spark handles nullability, a non-nullable...
The FAQ has been updated with: Remember to copy files without changing the timestamps to ensure that the time travel with timestamps will be consistent. This needs to include a...
