astro-sdk

astro-sdk copied to clipboard

Set SQL project initialisation to also initialises Airflow db

1

Improve the project initialisation, so it also initialises an Airflow DB Improve the project's interface so commands used after init can reference the project directories & files at easy. Example...

tatiana

**Describe the bug** Currently we are pushing the values to xcom to get the values in `extract_on_complete` for open lineage. This is a temporary hack till the issue is not...

sunank200

**Please describe the feature you'd like to see** We should be able to extract open lineage info from the `MergeOperator`. **Describe the solution you'd like** - Add a method `get_openlineage_facets`...

kaxil

We should also a add a unit-test in a follow-up PR next week _Originally posted by @kaxil in https://github.com/astronomer/astro-sdk/pull/1065#discussion_r995837033_

kaxil

Add benchmarking config

2

# Description ## What is the current behavior? 1. We don't have the complete benchmarking config added to the main branch. 2. As per the earlier discussion we need to...

utkarsharma2

**Please describe the feature you'd like to see** In many places including lineage extractors we notice that we need to know the row count of a table so it would...

pankajastro

**Describe the bug** When running benchmarking we are observing a large amount of memory being consumed and not getting free. As a result, we need to use a large machine...

utkarsharma2

**Describe the bug** When inferring schema complete Parquet is loaded in memory. https://github.com/astronomer/astro-sdk/blob/e4111397387e43ba7c8d2c0f5645376c466d271a/python-sdk/src/astro/files/types/parquet.py#L45 **Version** * Astro: 1.1.1 **Expected behavior** Research - if there is a better way to read schema...

utkarsharma2

Disable the SDK from changing capitalization of output columns by default

1

**Describe the bug** Users expressed confusion on why the column names for output tables are lowercased. It's due to this [default setting](https://github.com/astronomer/astro-sdk/blob/main/python-sdk/src/astro/sql/operators/dataframe.py#L110). **Expected behavior** The default behavior should be to...

pgzmnk

SnowflakeFile processing using astro-sdk

1

**Please describe the feature you'd like to see** Implement support for SnowflakeFile through astro-sdk **Describe the solution you'd like** A clear and concise description of what you want to happen,...

phanikumv