oso icon indicating copy to clipboard operation
oso copied to clipboard
verified publisher icon opensource-observer

Epic: Improved Dagster data replication

Open ryscheng opened this issue 9 months ago • 5 comments

What is it?

We have a bunch of amazing Dagster data producing factories. We are going to need WAY more data than we have today.

This epic reflects a rewrite that improves both UX of adding new data sources, local testing, and the infrastructure for how it runs/orchestrated

  • Phase 1: We rewrite our factories to support different backends (e.g. duckdb)
  • Phase 2: We support local testing and dynamic seed generation
  • Phase 3: Any user on the OSO web app can configure a new data source and monitor its progress

ryscheng avatar Apr 01 '25 17:04 ryscheng

OSO-596 Epic: Dynamic Data Sources

linear[bot] avatar Apr 01 '25 17:04 linear[bot]

@IcaroG can you help fill in sub-issues for this?

ryscheng avatar Apr 01 '25 19:04 ryscheng

@IcaroG it's also worth considering whether we want to also add support to just dynamically add Trino connectors (without replicating via dlt).

ryscheng avatar Apr 02 '25 04:04 ryscheng

Going to deprioritize this for now as discussed, at least until we decide how to move forward on blockchain indexing

ryscheng avatar Jun 03 '25 21:06 ryscheng

@Jabolol can you triage this? In particular, we want to revisit Dagster straight to Iceberg

ryscheng avatar Oct 15 '25 23:10 ryscheng