great_expectations

great_expectations copied to clipboard

java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext

3

**Describe the bug** I am configuring a SimpleCheckpoint that should be used in combination with a Spark dataframe. I am bootstrapping this with the following code. ``` def getDataContext(store_root_dir: str)...

olemara

Add a Data Docs site to DataContext() with code

1

**Is your feature request related to a problem? Please describe.** I want to add a new `data_docs_site` for my `DataContext` object using code. (e.g: A Doc for Azure). Currently, we...

luizfgoncalves

create_temp_table parameter not used in SQLAlchemy execution_engine

2

**Describe the bug** Disabling temporary table creation via `create_temp_table` in either the `great_expectations.yml` file or `batch_spec_passthrough` is not leveraged for the SQLAlchemy `execution_engine`. **To Reproduce** Some code adjusted to hide...

carlsonp

Idea for checking if all distinct value of a foreign key of a table is included in its dimension table or not

4

**Is your feature request related to a problem? Please describe.** Problem statement: It is to check if all distinct values of a fact table's key column (says, product key) is...

derek-hk

ConfigNotFoundError when instantiating DataContext with context_root_dir

1

**Describe the bug** The DataContext allows instantiating from a user-defined [context_root_dir](https://github.com/great-expectations/great_expectations/blob/5adf91121d49ce3144a1b4ccaa5ac58c00a99373/great_expectations/data_context/data_context/data_context.py#L314). The in-code documentation indicates this should be [the parent folder containing the great_expectations folder](https://github.com/great-expectations/great_expectations/blob/5adf91121d49ce3144a1b4ccaa5ac58c00a99373/great_expectations/data_context/data_context/data_context.py#L342) However, [when constructing the path...

paololucchino

Unable to create User DSN as Datasource

4

When creating new datasource, the process did not seem to finish and got stuck. I had to press Ctrl+C to quit the process and no datasource was created. I copied...

tchung555

`UpdateDataDocsAction` takes too long

1

**Describe the bug** I was wondering why `UpdateDataDocsAction` takes so long. In my case, it takes ~20min to run this action. Currently, there are ~1200 runs already stored in the...

sp1thas

Arguments for custom connectors (plugins) are stripped from the config

2

**Describe the bug** The docs [describe](https://docs.greatexpectations.io/docs/terms/plugin/) that you can build your own custom connectors and include them in your config. When you create your own connector, you face 2 issues...

sdebruyn

Usage of boto3_options when connected to s3-compatible Yandex Object Storage (https://storage.yandexcloud.net)

14

**Describe the bug** I try to get batch from csv file stored on s3 using attached config (see config in **Additional context**). I've set environmet vars AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY and...

TkachevKonstantin

Spark Config not applied to spark session.

6

**Describe the bug** I have followed this tutorial https://docs.greatexpectations.io/en/latest/guides/how_to_guides/configuring_datasources/how_to_configure_a_self_managed_spark_datasource.html and in particular my datasource looks like this (or similar): ``` datasources: spark_dataframe: data_asset_type: class_name: SparkDFDataset module_name: great_expectations.dataset batch_kwargs_generators: spark_sql_query: class_name:...

Spooky-0