hooli-data-eng-pipelines

hooli-data-eng-pipelines copied to clipboard

[RFC] use pip sync instead of pip install for pinned dependencies

6

This PR introduces a way to separate the python packages you need vs. the pinned dependencies. ## The motivating factors to adopt this: * we can control when we upgrade...

cnolanminich

from the README: > NOTE: if you are running Sling locally, there is currently an error message if you already have a duckdb database set up the first time you...

cnolanminich

Example here: https://github.com/dagster-io/dagster-open-platform/tree/main

cnolanminich

example-show-changed-assets

8

this PR: * makes a change to a model * adds a new model * makes a change to a source this is to help showcase the power of diff...

cnolanminich

cnolanminich

- [x] Remove dbt2 - [ ] Remove DBT_PROFILES_DIR. Just place the profiles.yml in the root dbt project directory - [ ] Link resources to creating the manifest at build...

slopp

Add Airbyte

1

Add an example that uses: https://hooli-airbyte.purina.elementl.dev/ One consideration with this example is deciding how to include it such that the repository still runs locally with minimal dependencies. It may also...

slopp

add new fetch row_count to dbt_assets

4

example in local (duckdb) 🎉

cnolanminich

Update sling assets

3

Updating the Sling demo to the new APIs. Summary of updates: - added replication python config - updated SlingResource to use SlingConnectionResource instead of SlingSourceConnection/SlingTargetConnection (soon to be deprecated) -...

izzye84

Column level lineage example on source assets

1

Demonstrates creating a column-level lineage based on a source asset using the example from the docs.

cnolanminich