Ori Hoch

another possibility - use docker compose yaml files to setup the test environment for each db engine

disabled the relevant dag in airflow

> Adding local env vars see comment in #1787 regarding env vars, same comment applies here - the test and dev environments are essentially the same > Creating a specific...

yes, same way we handle all the other production secrets

The main goal is to improve the ability to make changes and to maintain the code in the long-term * Writing mid results to disk allows to split the full...

issue is done, ready for review Airflow server was deployed (ask @atalyaalon for credentials): * prod: https://airflow.anyway.co.il/ * dev: https://dev-airflow.anyway.co.il/ ETL data directory is availble at: * prod: https://airflow-data.anyway.co.il/ *...

the question will make more sense if it will be ordered like this: 

@microsoft-github-policy-service agree

> Hi @OriHoch, Thanks for this PR, it is much more comprehensive than mine. Just wanted to flag that I noticed that when I run this in a jupyter notebook...

> > > > I think any guidance call will produce this issue but you can add this code to a notebook and give it a try: > > ```...