metaflow
metaflow copied to clipboard
Expose more accurate task / run lifecycle via Client API
Today, you can use the Client API to see whether a task/run has completed successfully (including caught exceptions) through .finished and .successful attributes.
Through the current API, you are not able to see the state of runs that
- are scheduled or being started by a production orchestrator like Step Functions or Argo Workflows
- have crashed vs. that are still running.
It would be useful to expose more accurate task/run state information through the Client API, some of which is already visible in the Metaflow UI (through heartbeats).
This feature we really need, like spark connect server, https://spark.apache.org/docs/latest/spark-connect-overview.html spark provide API make IDE or Jupyterlab can submit job use API; It is very important。