runner icon indicating copy to clipboard operation
runner copied to clipboard

No message retrieved from session within last 30 minutes

Open denny60004 opened this issue 2 years ago • 2 comments

Describe the bug We use a self-hosted runner to execute our tasks, which usually take about an hour to complete. However, occasionally we encounter this error No message retrieved from session 'c91ab00a-8cb1-45e0-a09c-5aced4c5889b' within last 30 minutes.. Whenever this error occurs, the GitHub Actions job log gets stuck at "Waiting for a runner to pick up this job," and then the runner shuts down. We have monitored indicators such as CPU and memory, and there appears to be no signs of overload.

To Reproduce Steps to reproduce the behavior:

  1. Go to '...'
  2. Run '....'
  3. See error

Expected behavior The task should be executed and completed correctly.

Runner Version and Platform

Version of your runner? 2.311.0

OS of the machine running the runner? Linux

What's not working?

Please include error messages and screenshots.

Job Log Output

Requested labels: self-hosted, t3-xlarge
Waiting for a runner to pick up this job...

Runner and Worker's Diagnostic Logs

| 2023-12-26T05:19:15.501+08:00 | [2023-12-25 21:19:10Z INFO JobDispatcher] Successfully renew job request 47479, job is valid till 12/25/2023 21:29:10
| 2023-12-26T05:20:15.501+08:00 | [2023-12-25 21:20:10Z INFO JobDispatcher] Successfully renew job request 47479, job is valid till 12/25/2023 21:30:10
| 2023-12-26T05:21:15.502+08:00 | [2023-12-25 21:21:10Z INFO JobDispatcher] Successfully renew job request 47479, job is valid till 12/25/2023 21:31:10
| 2023-12-26T05:22:15.501+08:00 | [2023-12-25 21:22:11Z INFO JobDispatcher] Successfully renew job request 47479, job is valid till 12/25/2023 21:32:10
| 2023-12-26T05:23:16.501+08:00 | [2023-12-25 21:23:12Z INFO JobDispatcher] Successfully renew job request 47479, job is valid till 12/25/2023 21:33:11
| 2023-12-26T05:24:17.501+08:00 | [2023-12-25 21:24:12Z INFO JobDispatcher] Successfully renew job request 47479, job is valid till 12/25/2023 21:34:12
| 2023-12-26T05:25:17.502+08:00 | [2023-12-25 21:25:12Z INFO JobDispatcher] Successfully renew job request 47479, job is valid till 12/25/2023 21:35:12
| 2023-12-26T05:26:17.502+08:00 | [2023-12-25 21:26:12Z INFO JobDispatcher] Successfully renew job request 47479, job is valid till 12/25/2023 21:36:12
| 2023-12-26T05:27:17.502+08:00 | [2023-12-25 21:27:13Z INFO JobDispatcher] Successfully renew job request 47479, job is valid till 12/25/2023 21:37:12
| 2023-12-26T05:28:18.502+08:00 | [2023-12-25 21:28:13Z INFO JobDispatcher] Successfully renew job request 47479, job is valid till 12/25/2023 21:38:13
| 2023-12-26T05:29:18.501+08:00 | [2023-12-25 21:29:13Z INFO JobDispatcher] Successfully renew job request 47479, job is valid till 12/25/2023 21:39:13
| 2023-12-26T05:30:19.501+08:00 | [2023-12-25 21:30:14Z INFO JobDispatcher] Successfully renew job request 47479, job is valid till 12/25/2023 21:40:14
| 2023-12-26T05:31:19.502+08:00 | [2023-12-25 21:31:14Z INFO JobDispatcher] Successfully renew job request 47479, job is valid till 12/25/2023 21:41:14
| 2023-12-26T05:32:19.502+08:00 | [2023-12-25 21:32:15Z INFO JobDispatcher] Successfully renew job request 47479, job is valid till 12/25/2023 21:42:14
| 2023-12-26T05:33:20.501+08:00 | [2023-12-25 21:33:15Z INFO JobDispatcher] Successfully renew job request 47479, job is valid till 12/25/2023 21:43:15
| 2023-12-26T05:34:20.502+08:00 | [2023-12-25 21:34:15Z INFO JobDispatcher] Successfully renew job request 47479, job is valid till 12/25/2023 21:44:15
| 2023-12-26T05:35:17.627+08:00 | [2023-12-25 21:35:15Z INFO JobDispatcher] Successfully renew job request 47479, job is valid till 12/25/2023 21:45:15
| 2023-12-26T05:35:22.501+08:00 | [2023-12-25 21:35:17Z INFO MessageListener] No message retrieved from session 'c91ab00a-8cb1-45e0-a09c-5aced4c5889b' within last 30 minutes.

denny60004 avatar Dec 26 '23 07:12 denny60004

we got same issue for Windows. Any workaround or fix?

vhuang-cnx avatar Mar 27 '24 15:03 vhuang-cnx

we got same issue for Windows. Any workaround or fix?

Since we are using GitHub's webhook to trigger the action runner's auto scaling, our temporary solution is to resend the webhook with jobs still in pending status, to retrigger it.

denny60004 avatar Apr 03 '24 03:04 denny60004