metaflow icon indicating copy to clipboard operation
metaflow copied to clipboard

Expose more accurate task / run lifecycle via Client API

Open tuulos opened this issue 2 years ago • 1 comments

Today, you can use the Client API to see whether a task/run has completed successfully (including caught exceptions) through .finished and .successful attributes.

Through the current API, you are not able to see the state of runs that

  • are scheduled or being started by a production orchestrator like Step Functions or Argo Workflows
  • have crashed vs. that are still running.

It would be useful to expose more accurate task/run state information through the Client API, some of which is already visible in the Metaflow UI (through heartbeats).

tuulos avatar Aug 29 '23 16:08 tuulos

This feature we really need, like spark connect server, https://spark.apache.org/docs/latest/spark-connect-overview.html spark provide API make IDE or Jupyterlab can submit job use API; It is very important。

jixianyihao avatar Dec 20 '23 03:12 jixianyihao