delta
delta copied to clipboard
delta-io
Roadmap 2022 H2 (discussion)
This is a working issue for folks to provide feedback on the prioritization of the Delta Lake priorities spanning July to December 2022. With the release of Delta Lake 2.0, we wanted to take the opportunity to discuss other vital features for prioritization with the community based on the feedback from the Delta Users Slack, Google Groups, Community AMAs (on Delta Lake YouTube), the Roadmap 2022H2 (discussion), and more.
Priority 0
We will focus on these issues and continue to deliver parts (or all of the issue) over the next six months
| Issue | Category | Task | Description |
|---|---|---|---|
| ~https://github.com/delta-io/connectors/pull/256~ [DONE] | Flink | Flink Source | Build Flink source to read Delta tables in batch and streaming jobs |
| https://github.com/delta-io/connectors/issues/238 | Flink | Flink SQL+ Table API + Catalog Support | After Flink Sink and Source, build support for Flink Catalog, SQL, and Table API |
| https://github.com/delta-io/connectors/issues/411, https://github.com/delta-io/connectors/issues/410 | Flink | Productionize support for all cloud object stores | Make sure that Flink Sink can write robustly to S3, GCS, ADLS2 with full transactional guarantees |
| Rust | Integrate with a common object-store abstraction from arrow / Rust ecosystem | This will allow us to provide a more convenient and performant API on the Rust and python side | |
| Rust | Support V2 writer protocol | Utilize PyArrow-based writer function (write_deltalake) support writer protocol V2 and object stores S3, GCS, and ADLS2. | |
| Rust | Expand write support for cloud object stores | Write to object stores S3, GCS, and ADLS2 from multiple clusters with full transactional guarantees | |
| ~#1257~ [DONE] | Spark | Release Delta 2.1 on Apache Spark 3.3 | Ensure the latest version of Delta Lake works with the latest version of Apache Spark™ |
| #1367 | Spark | Support reading tables with Deletion Vectors | Allow reads on tables that have deletion vectors to mark rows in parquet files as removed. |
| ~#1242~ [DONE] | Spark | Support time travel SQL syntax | Delta currently supports time travel via Python and Scala APIs. We would like to extend support for the SQL syntax VERSION AS OF and TIMESTAMP AS OF in SELECT statements. |
| Standalone | Extend Delta Standalone for higher protocol versions | Extend Delta Standalone to support logs using higher protocol versions and advanced features like constraints, generated columns, column mapping, etc. | |
| Standalone | Expand support for data skipping in Delta Standalone | Extend the current data skipping to skip file using column stats and more expressions | |
| Website | Updated Delta Lake documentation | Move Delta Lake documentation to the website GitHub repo to allow easier community collaboration | |
| Website | Consolidate all connector documentation | Consolidate docs of all connectors in the website Github repo |
Priority 1
We should be able to deliver parts (or all of the issue) over the next six months
| Issue | Category | Task | Description |
|---|---|---|---|
| Flink | Column stats generation in Flink Sink | Make the Flink Delta sink generate column stats | |
| Core | Delta Acceptance Testing (DAT) | With various languages interacting with the Delta protocol (e.g., Delta Standalone, Delta Spark, Delta Rust, Trino, etc.), we propose to have the same reference tables and library of reference tests to ensure all Delta APIs remain in compliance. | |
| Delta connectors | GoLang Delta connector | Support GoLang reading a Delta Lake table natively | |
| Delta connectors | Improve partition filtering in Power BI client | Improved partition filtering using built-in UI filters in Power BI | |
| Delta connectors | Pulsar Source connector | Support Apache Pulsar reading a Delta Lake table natively | |
| Presto/Trino | Support higher protocol versions in Presto and Trino | Use Standalone to support higher protocol versions | |
| Rust | Delta Rust API Updates | Update APIs and support more high-level operations on top of delta; this includes better conflict resolution | |
| Rust | Better support for large logs | Better support for handling large Delta logs/snapshots | |
| Sharing Connectors | GoLang Delta Sharing client | Support GoLang client for Delta Sharing | |
| Sharing Connectors | R Delta Sharing client | Support R client for Delta Sharing | |
| #1072 | Spark | Support for Identity columns | Create an identity column that will be automatically assigned a unique and statistically increasing (or decreasing if the step is negative) value. |
| Spark | Support querying Change Data Feed (CDF) using SQL queries | To support querying CDF using SQL queries in Apache Spark, we need to allow custom TVFs to be resolved using injected rules. | |
| ~With batch CDF~ [DONE in Delta 2.1] | Spark | Improve semantics of column mapping and Change Data Feed | Improve semantics of how column renames/drops (aka column mapping) interact with CDF and streaming |
Priority 2
Nice to have
| Issue | Category | Task | Description |
|---|---|---|---|
| Sharing | Share individual partitions | Support Sharing individual partitions in Delta Sharing | |
| Sharing Connectors | Rust Delta Sharing client | Support Rust client for Delta Sharing | |
| Sharing Connectors | Starburst/Trino Delta Sharing connector | Support Starburst/Trino client for Delta Sharing | |
| Sharing Connectors | Airflow Delta Sharing connector | Support sharing data from Airflow sensor |
History
- 2022-08-01: Initial creation
- 2022-08-02: Delta Sharing updates
- 2022-08-08: Include Identity columns in the roadmap
Note, we will be adding/updating the issue over the next few weeks but I'm a little behind schedule so thought I would get the roadmap discussion started ASAP. Thanks!
Hi folks. Can't see it explicitly mentioned so thought I'd ask - will identity columns support (i.e. writer version 6) be added in this H2 wave? That's a big feature we're keen to be able to use outside of Databricks, and it didn't quite make it into 2.0 by the looks of things.
Hi folks. Can't see it explicitly mentioned so thought I'd ask - will identity columns support (i.e. writer version 6) be added in this H2 wave? That's a big feature we're keen to be able to use outside of Databricks, and it didn't quite make it into 2.0 by the looks of things.
Thanks for the call out @edfreeman - identity columns has been added :)
Hi @dennyglee, what about Auto compaction and Optimize Write? I don't think the PRs are getting some attention for review / merge. Could you add it to the roadmap?
Oh good point! Let me get back to you on this shortly! Sorry about that!
On Wed, Aug 17, 2022 at 18:41 EJ Song @.***> wrote:
Hi @dennyglee https://github.com/dennyglee, what about Auto compaction and Optimize Write? I don't think the PRs are getting some attention for review / merge. Could you add it to the roadmap?
— Reply to this email directly, view it on GitHub https://github.com/delta-io/delta/issues/1307#issuecomment-1218895025, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALBHLLDOCHFYXAGKWGKXEDVZWIC3ANCNFSM55JXKP7A . You are receiving this because you were mentioned.Message ID: @.***>
Hi @dennyglee, what about support for displaying DDL of delta tables (SHOW CREATE TABLE)
https://github.com/delta-io/delta/issues/1032
https://github.com/delta-io/delta/pull/1255
Hi @dennyglee, what about support for displaying DDL of delta tables (
SHOW CREATE TABLE) #1032 #1255
Good call out @keen85 - let me check with @zpappa on this!
Hi @dennyglee, what about support for displaying DDL of delta tables (
SHOW CREATE TABLE) #1032 #1255Good call out @keen85 - let me check with @zpappa on this!
I have some minor style and test updates for this PR to be considered done, I can finish them today and we can try to pull them Into 2.1
How about delta caching that is present on databricks? Is there a plan for such feature?
"Delta caching" is actually a Databricks Runtime engine feature, not part of the format. Caching data on an processing engine's executor/workers nodes is something that can really be done well by the engine itself, not by a data format. It's unfortunate and confusing that we had marketed it under the "Delta" brand name, even though it's really not part of the "Delta Lake" storage format. So, in short, its not really possible to open source that as part of Delta Lake.
I Have being experimenting with Delta lake in Google Cloud and DuckDB and it is very promising, but without a local SSD cache it will never be fast enough, maybe we need a cache for Delta independently from Databricks implementation, Delta knows which files needs to be scanned, keeping a local copy on the first call will be really useful, at least for a the standalone reader
Sorry about that @sezruby - yes, we will be adding these to the roadmap very shortly. Thanks for your patience (I’ve been out the last two weeks)
Some quick updates:
- @sezruby Thanks for your patience - auto compaction and optimize writes have been included to the roadmap.
- @khwj included bloom filters and added some comments directly to your issue
- @MaksGS09 We're still determining the resources required for performant BQ integration
- We are currently reviewing #1255 and #1377
HTH!
Hi @dennyglee ! I know it was on the 2022 H1 github page, but I haven't seen any mention on clone functionality being moved into the open source library. Is there any update around that? I poked around the current source code but didn't really see it anywhere.
Thanks @p2bauer - great call out. I've added this to the roadmap and created issue #1387 to track this. HTH!
Hi @dennyglee , Is there any update on supporting higher protocol versions in Presto and Trino?
Great question @SanthoshPrasad - we've been working with the PrestoDB and the Trino communities on this and we should have some updates on various progress around this over the next couple of months. One of the methods we're doing this is through our DAT effort (Delta Acceptance Testing) so we can more cleanly document and clarify which APIs are on which protocol version. If you're interested in learning more on this, please join us in the #dat channel in Delta Users Slack. HTH!
Suggest we add Airbyte Destination S3: add delta lake/delta table support to the roadmap as it's already part of the Delta Rust Roadmap - WDYT?
I'd like to suggest adding "Register VACUUM in delta log" to the roadmap
- https://github.com/delta-io/delta/issues/868
- https://github.com/delta-io/delta/pull/1451
I know that each commit, min/max values are calculated for each parquet file and are present in the delta log json, but how about adding more granularity to existing data skipping mechanism, by using parquet page skipping? Relevant links:
- https://issues.apache.org/jira/browse/PARQUET-922
- https://blog.cloudera.com/speeding-up-select-queries-with-parquet-page-indexes/
Would this be doable?
I know that each commit, min/max values are calculated for each parquet file and are present in the delta log json, but how about adding more granularity to existing data skipping mechanism, by using parquet page skipping? Relevant links:
- https://issues.apache.org/jira/browse/PARQUET-922
- https://blog.cloudera.com/speeding-up-select-queries-with-parquet-page-indexes/
Would this be doable?
@dudzicp Oh, could you please create a separate issue for this and we can discuss the specifics there? Thanks!
Hey, I am interested in more details regarding https://delta.io/sharing/ It's stated that presto and trino are coming soon, but I could not really find any details or timelines. Please notice, that I am asking regaring delta sharing in context of Uniyt Catalog in particular and not necessarily regarding delta & trino/presto integration
Ahh, for Delta Sharing features within the context of UC, please ping Databricks community. Thanks!
Hey @dennyglee any updates on the Roadmap? :) I was creating some issues in the mack project (e.g. Python support for table property update) but wanted to make sure that the delta-spark team is not working on the things that I came up with already.
Thanks for your patience @robertkossendey - we're working on this but admittedly way behind schedule due to all of the various asks, eh?! Saying this, please continue working on mack project activities as those are the ones we're pretty sure make more sense for mack to address or at least if we plan to merge this into delta-spark, it'll be further out on the roadmap. Thanks for the ping, eh?!