Running metaflow-service and UI locally sees artifacts and DAG not displayable

[charlesdong1991]

Setting up metaflow-service and UI locally and run an example flow, I can see the Flow information. However, all artifacts are NOT displayed, see 'NoneType' object has no attribute 'get'. But locally on notebook, I am able to get the artifacts, e.g. Task('Flow/8/start/23', attempt=0)['df'].data.

Having a look at the chrom inspect, I can see a 500 internal error related to CORS:

Request URL: http://localhost:8083/flows/RevenuePredictionFlow/runs/8/dag
Request Method: GET
Status Code: 500 Internal Server Error
Remote Address: [::1]:8083
Referrer Policy: strict-origin-when-cross-origin

But in the code, i can see there is loose setting for it already: https://github.com/Netflix/metaflow-service/blob/9c1d587e0e88783fcaeb367dbebb624675311238/services/utils/init.py#L119

I think this results in the artifacts not displayable. Can anyone help with that and point out where i might do wrong?

Thank you!

[charlesdong1991]

Diving into the error log, i can see:

Traceback (most recent call last):
File \"/usr/local/lib/python3.7/runpy.py\", line 193, in _run_module_as_main \n \"__main__\", mod_spec)
File \"/usr/local/lib/python3.7/runpy.py\", line 85, in _run_code    
exec(code, run_globals)
File \"/root/services/ui_backend_service/data/cache/client/cache_server.py\", line 307, in <module> cli(auto_envvar_prefix='MFCACHE')
File \"/usr/local/lib/python3.7/site-packages/click/core.py\", line 1128, in __call__\n    return self.main(*args, **kwargs)
File \"/usr/local/lib/python3.7/site-packages/click/core.py\", line 1053, in main\n    rv = self.invoke(ctx)
File \"/usr/local/lib/python3.7/site-packages/click/core.py\", line 1395, in invoke\n    return ctx.invoke(self.callback, **ctx.params)
File \"/usr/local/lib/python3.7/site-packages/click/core.py\", line 754, in invoke\n    return __callback(*args, **kwargs)
File \"/root/services/ui_backend_service/data/cache/client/cache_server.py\", line 301, in cli\n    Scheduler(store, max_actions).loop()
File \"/root/services/ui_backend_service/data/cache/client/cache_server.py\", line 199, in __init__\n    maxtasksperchild=512,  # Recycle each worker once 512 tasks have been completed
File \"/usr/local/lib/python3.7/multiprocessing/context.py\", line 119, in Pool\n    context=self.get_context())
File \"/usr/local/lib/python3.7/multiprocessing/pool.py\", line 176, in __init__\n    self._repopulate_pool()
File \"/usr/local/lib/python3.7/multiprocessing/pool.py\", line 241, in _repopulate_pool\n    w.start()
File \"/usr/local/lib/python3.7/multiprocessing/process.py\", line 112, in start\n    self._popen = self._Popen(self)
File \"/usr/local/lib/python3.7/multiprocessing/context.py\", line 277, in _Popen\n    return Popen(process_obj)
File \"/usr/local/lib/python3.7/multiprocessing/popen_fork.py\", line 20, in __init__\n    self._launch(process_obj)
File \"/usr/local/lib/python3.7/multiprocessing/popen_fork.py\", line 74, in _launch\n    code = process_obj._bootstrap()
File \"/usr/local/lib/python3.7/multiprocessing/process.py\", line 297, in _bootstrap\n    self.run()
File \"/usr/local/lib/python3.7/multiprocessing/process.py\", line 99, in run\n    self._target(*self._args, **self._kwargs)
File \"/usr/local/lib/python3.7/multiprocessing/pool.py\", line 121, in worker\n    result = (True, func(*args, **kwds))
File \"/root/services/ui_backend_service/data/cache/client/cache_worker.py\", line 29, in execute_action\n    execute(tempdir, action_cls, request)
File \"/root/services/ui_backend_service/data/cache/client/cache_worker.py\", line 56, in execute\n    invalidate_cache=req.get('invalidate_cache', False))
File \"/root/services/ui_backend_service/data/cache/generate_dag_action.py\", line 97, in execute\n    results[result_key] = json.dumps(dag)
File \"/usr/local/lib/python3.7/contextlib.py\", line 130, in __exit__\n    self.gen.throw(type, value, traceback)
File \"/root/services/ui_backend_service/data/cache/utils.py\", line 130, in streamed_errors\n    get_traceback_str()
File \"/root/services/ui_backend_service/data/cache/utils.py\", line 124, in streamed_errors\n    yield
File \"/root/services/ui_backend_service/data/cache/generate_dag_action.py\", line 93, in execute\n    dag = DataArtifact(\"{}/_graph_info\".format(param_step.task.pathspec)).data
File \"/usr/local/lib/python3.7/site-packages/metaflow/client/core.py\", line 904, in data\n    return filecache.get_artifact(ds_type, location[6:], meta, *components)
File \"/usr/local/lib/python3.7/site-packages/metaflow/client/filecache.py\", line 214, in get_artifact\n    [name],
File \"/usr/local/lib/python3.7/site-packages/metaflow/datastore/task_datastore.py\", line 361, in load_artifacts\n    for (key, blob) in self._ca_store.load_blobs(to_load.keys()):
File \"/usr/local/lib/python3.7/site-packages/metaflow/datastore/content_addressed_store.py\", line 140, in load_blobs\n    with open(file_path, \"rb\") as f:\n\nTypeError: expected str, bytes or os.PathLike object, not NoneType\n"

could it be that either metaflow-service or metaflow made some changes and caused to sort of conflicts in paths? So that it doesn't find it?

[charlesdong1991]

The same error is seen in DAG section.

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 307, in <module>
    cli(auto_envvar_prefix='MFCACHE')
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 301, in cli
    Scheduler(store, max_actions).loop()
  File "/root/services/ui_backend_service/data/cache/client/cache_server.py", line 199, in __init__
    maxtasksperchild=512,  # Recycle each worker once 512 tasks have been completed
  File "/usr/local/lib/python3.7/multiprocessing/context.py", line 119, in Pool
    context=self.get_context())
  File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 176, in __init__
    self._repopulate_pool()
  File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 241, in _repopulate_pool
    w.start()
  File "/usr/local/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/usr/local/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/usr/local/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/usr/local/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
    code = process_obj._bootstrap()
  File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/root/services/ui_backend_service/data/cache/client/cache_worker.py", line 29, in execute_action
    execute(tempdir, action_cls, request)
  File "/root/services/ui_backend_service/data/cache/client/cache_worker.py", line 56, in execute
    invalidate_cache=req.get('invalidate_cache', False))
  File "/root/services/ui_backend_service/data/cache/generate_dag_action.py", line 97, in execute
    results[result_key] = json.dumps(dag)
  File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/root/services/ui_backend_service/data/cache/utils.py", line 130, in streamed_errors
    get_traceback_str()
  File "/root/services/ui_backend_service/data/cache/utils.py", line 124, in streamed_errors
    yield
  File "/root/services/ui_backend_service/data/cache/generate_dag_action.py", line 93, in execute
    dag = DataArtifact("{}/_graph_info".format(param_step.task.pathspec)).data
  File "/usr/local/lib/python3.7/site-packages/metaflow/client/core.py", line 904, in data
    return filecache.get_artifact(ds_type, location[6:], meta, *components)
  File "/usr/local/lib/python3.7/site-packages/metaflow/client/filecache.py", line 214, in get_artifact
    [name],
  File "/usr/local/lib/python3.7/site-packages/metaflow/datastore/task_datastore.py", line 361, in load_artifacts
    for (key, blob) in self._ca_store.load_blobs(to_load.keys()):
  File "/usr/local/lib/python3.7/site-packages/metaflow/datastore/content_addressed_store.py", line 140, in load_blobs
    with open(file_path, "rb") as f:

TypeError: expected str, bytes or os.PathLike object, not NoneType