metaflow icon indicating copy to clipboard operation
metaflow copied to clipboard

[Ready for Review] fix bug where client can not access foreach stack

Open darinyu opened this issue 1 year ago • 3 comments

To reproduce, have a flow with foreach task, and then in python client, do

from metaflow import Task
>>> task = Task("ForeachFlow/123/foreach_step/task-00000000")
>>> task.index
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/metaflow/metaflow/client/core.py", line 1166, in index
    return self["_foreach_stack"].data[-1].index
  File "/root/metaflow/metaflow/client/core.py", line 875, in data
    obj = filecache.get_artifact(ds_type, location[6:], meta, *components)
  File "/root/metaflow/metaflow/client/filecache.py", line 207, in get_artifact
    _, obj = next(
  File "/root/metaflow/metaflow/datastore/task_datastore.py", line 370, in load_artifacts
    yield name, pickle.loads(blob)
AttributeError: Can't get attribute 'ForeachFrame' on <module '__main__' (built-in)>

This is because this ForeachFrame is loaded to main, and defined in task.py. Without a flow, this ForeachFrame is not imported. This PR refactors the namedtuple util to top level (with minimum dependency) and import it at top level.

darinyu avatar Mar 14 '24 22:03 darinyu

@darinyu -- is this no longer something you want merged? I thought it was still relevant.

romain-intel avatar Apr 11 '24 16:04 romain-intel

@darinyu for when you are back :)

romain-intel avatar Apr 30 '24 07:04 romain-intel

Yeah this still needs to go in, it slipped during before my vacation. I just rebased this commit and I think it is ready to go in now.

darinyu avatar May 09 '24 04:05 darinyu