dagster icon indicating copy to clipboard operation
dagster copied to clipboard

Cannot access `metadata_by_output_name` with op/graph-backed assets in io manager

Open erinov1 opened this issue 2 years ago • 1 comments

Dagster version

0.15.7

What's the issue?

I commented on https://github.com/dagster-io/dagster/pull/8861, but I'll file this here in case it's a bug. I was having trouble accessing metadata set using metadata_by_output_name in an io_manager, e.g., the following does not work:

from dagster import (
    AssetsDefinition,
    IOManager,
    io_manager,
    op,
    with_resources,
)


class MyIOManager(IOManager):
    def handle_output(self, context, obj):
        assert context.metadata.get("metadata") == "my_value"

    def load_input(self, context):
        pass


@io_manager
def my_io_manager():
    return MyIOManager()


@op
def my_op():
    pass


my_asset = AssetsDefinition.from_op(
    op_def=my_op, metadata_by_output_name={"result": {"metadata": "my_value"}}
)

my_asset_with_io_manager = with_resources(
    [my_asset], resource_defs={"io_manager": my_io_manager}
)

On the other hand, if I explicitly set the metadata in the op then this works fine, e.g.,

@op(out=Out(metadata={"metadata": "my_value"}))
def my_op():
    ...

Is this expected? It would nice to be able to access the metadata when parametrizing different outputs for assets with the same underlying op.

What did you expect to happen?

No response

How to reproduce?

No response

Deployment type

No response

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

erinov1 avatar Aug 04 '22 22:08 erinov1

Hi @erinov1 ! Thanks for the report, this is related to the issue here: https://github.com/dagster-io/dagster/issues/9076. I believe this issue can be fixed with a smaller change than the prior issue, so I'll see if we can get a fix out for this soon. Really appreciate the detailed summary!

OwenKephart avatar Aug 04 '22 22:08 OwenKephart