[CDAP-19455] Transition runs throttled by RuntimeJobManager to "rejected" state instead of "failed"

[arjan-bal]

CDAP 19455

When Dataproc submit job API is called, it returns a 429 HTTP code or RESOURCE_EXHAUSTED grpc code when their master agent isn't running on Dataproc master node. This transitions the pipeline to a failed state. Rejected state better represents rate limiting errors, so this PR aims to transition such failures to rejected state instead.

A new exception TooManyRequestsException is added to the runtime SPI. When runtime job managers throw this exception, pipeline is transitioned to rejected state by RemoteExecutionTwillRunnerService.

Added a function call to handle pipeline completion when pipeline transitions to rejected state. Previously transitions to rejected happened before reaching the starting state. In such cases call to handle pipeline completion is avoided.

Manually tested to ensure pipelines with unhealthy dataproc master node are rejected:

Manually tested to ensure flow control still works as it used to.

[gitpod-io[bot]]