mykola.yesypchuk

[Spark] expect_column_pair_values_A_to_be_greater_than_B(parse_strings_as_datetimes=True) is not working properly

1

https://github.com/great-expectations/great_expectations/blob/1f2bc4d3c33faa898f76ecd3bf267bd788bae9bc/great_expectations/dataset/sparkdf_dataset.py#L1362 Data: "col_a": ["07/11/1967", "04/07/1966", "01/03/1953"], "col_b": ["08/10/1960", "03/07/1966", "01/01/1900"], Engine: Spark Command: validator.expect_column_pair_values_A_to_be_greater_than_B("col_b", "col_a", parse_strings_as_datetimes=True) Expected: expectation result=Failed Now: expectation result=True Details: When we compare dates fields(string fields in...

AWS S3 nested path with spark-lineage issue

4

**Affected module** Ingestion Framework: spark-lineage https://docs.open-metadata.org/v1.3.x/connectors/ingestion/lineage/spark-lineage#configuration **Describe the bug** Failed to write data using spark and spark-lineage configuration to nested path like `s3a://test_bucket/data/group=db1/source_name=sp_entity` Error: 500 HTTP error in server API...

S3 Storage structureFormat parquet issue

2

**Affected module** Ingestion Framework **Describe the bug** Failed to run S3 storage metadata ingestion due _SUCCESS file in dataPath entries folder. **To Reproduce** openmetadata.json ``` { "entries": [ { "dataPath":...