Add max running time as an optional stop criteria

[sperlingxx]

/kind feature

Describe the solution you'd like Currently, katib has three kinds of succeed reason:

1. ExperimentGoalReached
2. ExperimentMaxTrialsReached
3. ExperimentSuggestionEndReached

In my opinion, it will bring a lot convenient to some users if we add another succeed criteria: max running time. We can add a new field MaxRunningTimeInMinutes *uint64 in experiment spec, and judge whether experiment duration exceeds pre-configured maximum running time.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

Label	Probability
feature	0.98

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

[gaocegege]

LGTM.

/priority p2

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[tenzen-y]

/reopen /lifecycle frozen

[google-oss-prow[bot]]

@tenzen-y: Reopened this issue.

In response to this:

/reopen /lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.