connectors

connectors copied to clipboard

Cache the partition pruning result in FilteredDeltaScanImpl to improve performance (#399) Signed-off-by: sonhmai

sonhmai

Performance Improvement: cache per-partition results during partition pruning

3

We perform partition pruning inside of [FilteredDeltaScanImpl](https://github.com/delta-io/connectors/blob/master/standalone/src/main/scala/io/delta/standalone/internal/scan/FilteredDeltaScanImpl.scala) by - having some input query / expression `expr` - creating a [PartitionRowRecord](https://github.com/delta-io/connectors/blob/master/standalone/src/main/scala/io/delta/standalone/internal/data/PartitionRowRecord.scala) for a given AddFile `partitionRowRecord` - evaluating `partitionRowRecord` against `expr`...

scottsand-db

Column stats based file pruning: expression type support p1

1

Support following expression type in [column stats based file pruning](https://docs.google.com/document/d/1D2ukbGInriKTOZJl3r3pCZoQRji40rSuFgnOyzhHz7k/edit). * IsNull * IsNotNull * Not

6a0juu

Support following expression type in [column stats based file pruning](https://docs.google.com/document/d/1D2ukbGInriKTOZJl3r3pCZoQRji40rSuFgnOyzhHz7k/edit). * equalTo * greaterThan * greaterThanOrEqual * lessThan * lessThanOrEqual * OR * AND

6a0juu

Performance Improvement: column stats based file pruning for Delta Standalone Reader.

4

[FilteredDeltaScan](https://github.com/delta-io/connectors/blob/master/standalone/src/main/scala/io/delta/standalone/internal/scan/FilteredDeltaScanImpl.scala) in Standalone now implemented partition filter, it is to: * having some input query / expression expr. * creating a [PartitionRowRecord](https://github.com/delta-io/connectors/blob/master/standalone/src/main/scala/io/delta/standalone/internal/data/PartitionRowRecord.scala) for a given AddFile partitionRowRecord. * evaluating partitionRowRecord...

6a0juu

Publishing empty source jars for `delta-standalone`

2

If you look at the source jars that are being published for delta standalone, the source jars are just empty. For example you can see this in the latest `io.delta:delta-standalone_2.13:0.4.1`...

ckipp01

Flink Delta Sink - Table API UPDATED

25

Adding Table/SQL support for Delta-Flink Sink connector. The Table architecture for Flink connectors is showed on below diagram:  This PR provides Delta specific implementation for: - DynamicTableSinkFactory - DynamitTableSink...

kristoffSC

FlinkDeltaSink: Add column stats from the parquet file to delta transaction log

14

Extracts the column level stats from parquet file and add it to the delta log in json format, fixes https://github.com/delta-io/connectors/issues/371 - The implementation is not complete as it may not...

gopik

DecimalType comparison expressions should be valid between values with decimal types with different scales/precisions

14

Currently our comparison expressions fail with `IllegalArgumentException` for decimal type arguments that don’t have equal precision/scale since we check for equal data types. This should be valid (and is in...

allisonport-db

Power BI Connector does not work when root path of delta table contained in function parameter

5

Hey there, when using power query to get the set of files for the delta connector, the following query results in an emtpy table when using the Power BI Delta...

TKlerx