metaflow-ui icon indicating copy to clipboard operation
metaflow-ui copied to clipboard
verified publisher icon Netflix

Crashed Flow appear to run forever.

Open mikejmills opened this issue 4 years ago • 2 comments

Description

I've had a few Flows which have crashed. The crash is due to my errors not metaflow itself. However in the UI it continues to be marked as "Running".

Steps to Reproduce

Not sure how to reproduce it on demand.

Expected behavior: Should be marked as failed

Actual behavior: Stays in a "Running" state.

Reproduces how often: 1 in 16 so far.

Versions

Additional Information

mikejmills avatar Oct 28 '21 21:10 mikejmills

A few questions:

  • are these flows run through step functions, locally or on batch?
  • which version of the metadata service are you running?

The logic should mark these flows as failed after some time (definitely not 1h -- usually less than a minute after they crash but it's configurable and depends on whether it was a sfn run or not).

romain-intel avatar Oct 29 '21 06:10 romain-intel

A few questions:

  • are these flows run through step functions, locally or on batch? These are run locally. No AWS Batch
  • which version of the metadata service are you running? 2.4.1

The logic should mark these flows as failed after some time (definitely not 1h -- usually less than a minute after they crash but it's configurable and depends on whether it was a sfn run or not).

mikejmills avatar Nov 01 '21 18:11 mikejmills