astro-sdk

astro-sdk copied to clipboard

kaxil

Enable Pylint

4

kaxil

Currently, deepsource fails for certain cases which we don't want to test. Example: It expects us to not have any comments including TODO, for instance. This Github issue involves ignoring...

kaxil

Try to change Astro Python SDK check example DAGs to use GreatExpecatations

1

**Context** We've been having discussions about the Astro Python SDK check strategy. It feels there is an overlap with Great Expectations. As part of this, we created: 1. https://github.com/astro-projects/astro/issues/298 2....

tatiana

Analyse the features available in Great Expectations and review the need for Astro custom checks

1

**Context** At the moment, Astro offers a few table checks (stats, boolean, aggregate). It may have an overlap with the Great Expectations package: https://github.com/great-expectations/airflow-provider-great-expectations We may want to check: https://github.com/astronomer/internal_data_quality/pull/7/files...

tatiana

Allow `aql.render` to use the DAG's `template_searchpath`

1

**Context** When instantiating Airflow DAGs, users can declare `template_searchpath`, which is a list of folders (non-relative) that define where jinja will look for templates. By default, it contains the DAGs...

tatiana

Issue in `load_file` some datasets in Snowflake

1

**Version:** `astro==0.4.0` **Problem** At the moment, we are unable to load the following dataset from Tate Gallery into Snowflake: https://github.com/tategallery/collection/blob/master/artwork_data.csv. The operation works using BQ and Postgres. I could not...

tatiana

In order to allow for easier local development AND support for storing dataframes in NFS, a user should be able to store intermediate values locally Acceptance criteria: - A user...

dimberman

The current system of loading is limited to the size of a single dataframe. This of course will not scale to full production use-cases. Proposed solution: 1. By default, we...

dimberman

**Context** At the moment the Astro Python SDK exposes three types of checks: * `AggregateCheck` * `BooleanCheck` * `StatsCheck` However, we have not written significant documentation about them. **Acceptance Criteria**...

tatiana