dagster icon indicating copy to clipboard operation
dagster copied to clipboard

[docs] - IO manager that operates on local file paths example

Open sryza opened this issue 3 years ago • 0 comments

What content is missing

Something like:

@solid(output_defs=[OutputDefinition(metadata={"gcs_path": "/a/b/c"})])
def my_solid(_):
    local_file_name = process_data()
    return local_file_name

class GCSIOManager(IOManager):
    def handle_output(self, context, obj):
        write_local_file_to_gcs(local_file_name=obj, gcs_path=context.metadata["gcs_path"])

    def load_input(self, context):
        local_file_name = make_temporary_file()
        download_gcs_file_to_local(
            local_file_name=local_file_name, gcs_path=context.upstream_output.metadata["gcs_path"]
        )
        return local_file_name

Type of the content

Main Concept: https://docs.dagster.io/concepts/io-management/io-managers#examples

(Optional) Anything in particular you want the docs to cover

(Optional) Target Date

Writer's Guide

  • Docs README
  • Templates to follow:
  • Examples for reference:

Message from the maintainers:

Are you looking for the same documentation content? Give it a :thumbsup:. We factor engagement into prioritization.

sryza avatar Apr 28 '21 00:04 sryza