Cloud Runner, K8s - "RPC Failure" (inconsistent, instability)

[frostebite]

Bug description

Log stream while running kubernetes job is occasionally interrupted, the job seems to be deleted, and the message "RPC Failure" is shown.

How to reproduce

Inconsistent, use cloud runner k8s for long running jobs. The end to end tests seem to produce this slightly more often.

This may relate to Google Kubernetes Engine - "AutoPilot" for kubernetes that I use to host the test environment.

Expected behavior

Log streams should be as reliable as possible.

Additional details

Remediation would include recommending async mode for inconsistent log stream platforms. The log would have to be stored within the job and uploaded for this route as we must parse the output to properly confirm if a job passes.