Sandy Ryza
Sandy Ryza
When an asset sensor observes a materialization that corresponds to a particular partition, it should kick off runs that are downstream of that partition.
In a notebook, or in an op, you might want to load an asset to inspect it. Imagine if you could do something like this: ``` @repo def repo(): ......
## What content is missing Something like: ``` @solid(output_defs=[OutputDefinition(metadata={"gcs_path": "/a/b/c"})]) def my_solid(_): local_file_name = process_data() return local_file_name class GCSIOManager(IOManager): def handle_output(self, context, obj): write_local_file_to_gcs(local_file_name=obj, gcs_path=context.metadata["gcs_path"]) def load_input(self, context): local_file_name =...
Reported here: https://dagster.slack.com/archives/C01U954MEER/p1649685707570159 ``` @root_input_manager( input_config_schema={ "filepath": Field(str, is_required=True), "sep": Field(str, default_value=",", is_required=False, description="Delimiter to use."), } ) def csv_to_df(context): return pd.read_csv( context.config["filepath"], sep=context.config.get("sep") ) ``` In this test, the...
### Summary & Motivation Fixes #4591. Open questions and reservations: - Is this the right name? What if we want to add something in the future that's union of two...
Requests: - https://dagster.slack.com/archives/C01U954MEER/p1663315890022549?thread_ts=1663253415.453339&cid=C01U954MEER - https://dagster.slack.com/archives/C01U954MEER/p1658350429504999?thread_ts=1658345537.413359&cid=C01U954MEER (includes user implementation)
### What's the issue or suggestion? Partitioned I/O managers are I/O managers that pull partition information off of the context to make decisions about where to store data. ### Additional...
### What's the issue or suggestion? all the patterns described here - link to https://docs.dagster.io/concepts/ops-jobs-graphs/ops#outputs- apply to `@asset`- decorated functions as well ### Additional information _No response_ ### Message from...
### Summary & Motivation ### How I Tested These Changes