dagster
dagster copied to clipboard
[docs] - IO manager that operates on local file paths example
What content is missing
Something like:
@solid(output_defs=[OutputDefinition(metadata={"gcs_path": "/a/b/c"})])
def my_solid(_):
local_file_name = process_data()
return local_file_name
class GCSIOManager(IOManager):
def handle_output(self, context, obj):
write_local_file_to_gcs(local_file_name=obj, gcs_path=context.metadata["gcs_path"])
def load_input(self, context):
local_file_name = make_temporary_file()
download_gcs_file_to_local(
local_file_name=local_file_name, gcs_path=context.upstream_output.metadata["gcs_path"]
)
return local_file_name
Type of the content
Main Concept: https://docs.dagster.io/concepts/io-management/io-managers#examples
(Optional) Anything in particular you want the docs to cover
(Optional) Target Date
Writer's Guide
- Docs README
- Templates to follow:
- Examples for reference:
Message from the maintainers:
Are you looking for the same documentation content? Give it a :thumbsup:. We factor engagement into prioritization.