metaflow
metaflow copied to clipboard
[Ready for Review] fix bug where client can not access foreach stack
To reproduce, have a flow with foreach task, and then in python client, do
from metaflow import Task
>>> task = Task("ForeachFlow/123/foreach_step/task-00000000")
>>> task.index
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/root/metaflow/metaflow/client/core.py", line 1166, in index
return self["_foreach_stack"].data[-1].index
File "/root/metaflow/metaflow/client/core.py", line 875, in data
obj = filecache.get_artifact(ds_type, location[6:], meta, *components)
File "/root/metaflow/metaflow/client/filecache.py", line 207, in get_artifact
_, obj = next(
File "/root/metaflow/metaflow/datastore/task_datastore.py", line 370, in load_artifacts
yield name, pickle.loads(blob)
AttributeError: Can't get attribute 'ForeachFrame' on <module '__main__' (built-in)>
This is because this ForeachFrame is loaded to main, and defined in task.py. Without a flow, this ForeachFrame is not imported. This PR refactors the namedtuple util to top level (with minimum dependency) and import it at top level.
@darinyu -- is this no longer something you want merged? I thought it was still relevant.
@darinyu for when you are back :)
Yeah this still needs to go in, it slipped during before my vacation. I just rebased this commit and I think it is ready to go in now.