Lorin Dawson

[DRAFT] Migrate unit tests to chispa

1

fixes #233 - migrate unit tests for dataframe equality to chispa - removed methods for schema equality checks since this is performed as part of `assert_df_equality` - combined `assertDataFramesEqual` and...

[Chispa](https://github.com/MrPowers/chispa) has near identical functionality to classes currently used to unit test the spark code used for tempo. However, Chispa has improved logging and error reporting for failed test cases.

Using generated columns, we could get some partition filtering on year, month, and/or day when using the ts_col for filtering at file scan. https://docs.delta.io/latest/delta-batch.html#use-generated-columns From the docs on Generated Columns:...

SparkSQL: Update L014 rule to not flag Delta Change Data Feed Session & Table Property

2

### Brief summary of the change made Adds support for a new table property that should not be lowercased. ### Are there any other side effects of this change that...

`tsdf.select("*")` throws Exception stating columns must be present when they are by nature of the projection

1

Current behavior produces the below Exception: `Exception: In TSDF's select statement original ts_col, partitionCols and seq_col_stub(optional) must be present` However by nature of `select("*")` they will be present in the...

Remove partition columns from Z Order optimization in `io.py`

2

In `io.py` we Z ORDER on `partitionCols + optimizationCols` when `useDeltaOpt` is True. Since we can partition prune without Z Ordering on partition columns, I believe it makes sense to...

## Changes Fixes #626 ## Tests - [ ] `make test` run locally - [ ] `make fmt` applied - [ ] relevant integration tests applied

**Description** Documentation for `databricks.sdk.service.iam.AccountGroupsAPI` accidentally calls `WorkspaceClient` instead of `AccountClient` in the *Usage:* examples **Reproduction** None, https://databricks-sdk-py.readthedocs.io/en/latest/account/iam/groups.html#a-groups-account-groups **Expected behavior** Documented *Usages:* should use the correct client to avoid confusion. **Is...

Add on_kill equivalent to Databricks SQL Hook to cancel timed out queries

6

The Databricks Provider did not implement a mechanism to cancel SQL queries submitted by DatabricksSqlHook. This led to data quality issues, where Airflow would report a cancellation due to timeout;...