training-operator
training-operator copied to clipboard
[Feature request] Exit code on failure
Feature Request
I would like to propose a feature to get the exit code of the chief pod via TFJob.
I know that the TFJob already takes care of whether-to-restart based on restartPolicy and the exit code internally.
Can we expose that information via TFJob? Currently we want to handle the restart externally and it seems it is hard to get that information via TFJob CRD.
@tonykim-moloco what info do you need via TFJob ? Can you elaborate more
TFJob currently only exposes a wrapped information - Succeeded | Created | Running | Failed
However, in failure cases, I would like to know the exit code about the pod to understand about why it failed.
For example, if a worker pod crashed due to OOM (exit code 137), I would like to look up TFJob to see that exit code was 137 on one of the workers.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Probably, we can support this request once we introduce batch/Job.
#1718
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.