mpgreg

load_file replacing existing tables fails if there are views on the tables

1

load_file(if_exists='replace') works the first time a DAG is run. Upstream tasks subsequently create views on that DAG. Rerunning the DAG fails as it cannot drop the existing tables because of...

Addes a new attribute and operator arg for database override and parses data_asset_name for either TABLE, SCHEMA.TABLE or DATABASE.SCHEMA.TABLE. Due to differences in how schema and database are used across...

Currently if there is a '.' in the asset name it assumed to be a schema.table. Should be able to pass database.schema.table. https://github.com/mpgreg/airflow-provider-great-expectations/blob/be4dfad4feea8ab758a64407ce5b0e6010ad4842/great_expectations_provider/operators/great_expectations.py#L230

Need to specify tokenization for content

1

https://github.com/astronomer/ask-astro/blob/c45487c7f12a9424dbe885580c687e35e30b7de4/airflow/include/data/schema.json#L54 Without specifying a tokenization scheme ingest will default to `word` as per https://weaviate.io/developers/weaviate/config-refs/schema#property-tokenization. This will split snake-case configuration parameters and environment variables treating underscore as whitespace. Example as per...

Need to set class schema for check_schema and create_schema tasks

1

https://github.com/astronomer/ask-astro/blob/c45487c7f12a9424dbe885580c687e35e30b7de4/airflow/dags/ingestion/ask-astro-load.py#L18C10-L18C10 The class awareness was added in https://github.com/astronomer/ask-astro/commit/c45487c7f12a9424dbe885580c687e35e30b7de4 but the json schema file still references "Docs". The `check_schema` and `create_schema` tasks will need some logic to update the class name.

Upsert logic needs to be redesigned

1

**Describe the bug** Current code is not atomic and is not batch safe. Also no roll-back. If a documents chunks span a batch the upsert will remove docs and reinsert...