supervisor_checks

supervisor_checks copied to clipboard

Using image python:3.10-slim curling localhost:4180/ping in the container returns an OK

root@008a65d1d6b8:/# curl localhost:4180/ping
OK

oauth.sh works fine without running the event listener.

[program:oauth]
command                 = /opt/oauth-proxy/oauth.sh
user                    = application
process_name            = %(program_name)s

[eventlistener:oauth_check]
command=/usr/local/bin/supervisor_http_check -n oauth -u /ping -t 30 -r 3 -p 4180
events=TICK_60
stdout_logfile          = /dev/stdout
stderr_logfile          = /dev/stderr

container_instance  | Traceback (most recent call last):
container_instance  |   File "/usr/local/bin/supervisor_http_check", line 8, in <module>
container_instance  | 2022-02-13 18:45:05,447 CRIT uncaptured python exception, closing channel <POutputDispatcher at 139876582247632 for <Subprocess at 139876582250656 with name oauth_check in state STARTING> (stderr)> (<class 'OSError'>:[Errno 29] Illegal seek [/usr/local/lib/python3.10/site-packages/supervisor/supervisord.py|runforever|218] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|handle_read_event|281] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|record_output|215] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|_log|184] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|info|327] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|log|345] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|emit|227] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|doRollover|264])
container_instance  | 2022-02-13 18:45:05,447 CRIT uncaptured python exception, closing channel <POutputDispatcher at 139876582247632 for <Subprocess at 139876582250656 with name oauth_check in state STARTING> (stderr)> (<class 'OSError'>:[Errno 29] Illegal seek [/usr/local/lib/python3.10/site-packages/supervisor/supervisord.py|runforever|218] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|handle_read_event|281] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|record_output|215] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|_log|184] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|info|327] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|log|345] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|emit|227] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|doRollover|264])
container_instance  |     sys.exit(main())
container_instance  |   File "/usr/local/lib/python3.10/site-packages/supervisor_checks/bin/http_check.py", line 86, in main
container_instance  |     return check_runner.CheckRunner(
container_instance  |   File "/usr/local/lib/python3.10/site-packages/supervisor_checks/check_runner.py", line 62, in __init__
container_instance  |     self._rpc_client = childutils.getRPCInterface(self._environment)
container_instance  |   File "/usr/local/lib/python3.10/site-packages/supervisor/childutils.py", line 21, in getRPCInterface
container_instance  |     return xmlrpclib.ServerProxy('http://127.0.0.1', getRPCTransport(env))
container_instance  |   File "/usr/local/lib/python3.10/site-packages/supervisor/childutils.py", line 15, in getRPCTransport
container_instance  | 2022-02-13 18:45:05,447 CRIT uncaptured python exception, closing channel <POutputDispatcher at 139876582247632 for <Subprocess at 139876582250656 with name oauth_check in state STARTING> (stderr)> (<class 'OSError'>:[Errno 29] Illegal seek [/usr/local/lib/python3.10/site-packages/supervisor/supervisord.py|runforever|218] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|handle_read_event|281] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|record_output|215] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|_log|184] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|info|327] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|log|345] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|emit|227] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|doRollover|264])
container_instance  | 2022-02-13 18:45:05,447 CRIT uncaptured python exception, closing channel <POutputDispatcher at 139876582247632 for <Subprocess at 139876582250656 with name oauth_check in state STARTING> (stderr)> (<class 'OSError'>:[Errno 29] Illegal seek [/usr/local/lib/python3.10/site-packages/supervisor/supervisord.py|runforever|218] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|handle_read_event|281] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|record_output|215] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|_log|184] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|info|327] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|log|345] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|emit|227] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|doRollover|264])
container_instance  |     return SupervisorTransport(u, p, env['SUPERVISOR_SERVER_URL'])
container_instance  |   File "/usr/local/lib/python3.10/os.py", line 679, in __getitem__
container_instance  | 2022-02-13 18:45:05,447 CRIT uncaptured python exception, closing channel <POutputDispatcher at 139876582247632 for <Subprocess at 139876582250656 with name oauth_check in state STARTING> (stderr)> (<class 'OSError'>:[Errno 29] Illegal seek [/usr/local/lib/python3.10/site-packages/supervisor/supervisord.py|runforever|218] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|handle_read_event|281] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|record_output|215] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|_log|184] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|info|327] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|log|345] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|emit|227] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|doRollover|264])
container_instance  | 2022-02-13 18:45:05,447 CRIT uncaptured python exception, closing channel <POutputDispatcher at 139876582247632 for <Subprocess at 139876582250656 with name oauth_check in state STARTING> (stderr)> (<class 'OSError'>:[Errno 29] Illegal seek [/usr/local/lib/python3.10/site-packages/supervisor/supervisord.py|runforever|218] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|handle_read_event|281] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|record_output|215] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|_log|184] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|info|327] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|log|345] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|emit|227] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|doRollover|264])
container_instance  |     raise KeyError(key) from None
container_instance  | KeyError: 'SUPERVISOR_SERVER_URL'
container_instance  | 2022-02-13 18:45:05,448 CRIT uncaptured python exception, closing channel <POutputDispatcher at 139876582247632 for <Subprocess at 139876582250656 with name oauth_check in state STARTING> (stderr)> (<class 'OSError'>:[Errno 29] Illegal seek [/usr/local/lib/python3.10/site-packages/supervisor/supervisord.py|runforever|218] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|handle_read_event|281] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|record_output|215] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|_log|184] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|info|327] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|log|345] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|emit|227] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|doRollover|264])
container_instance  | 2022-02-13 18:45:05,448 CRIT uncaptured python exception, closing channel <POutputDispatcher at 139876582247632 for <Subprocess at 139876582250656 with name oauth_check in state STARTING> (stderr)> (<class 'OSError'>:[Errno 29] Illegal seek [/usr/local/lib/python3.10/site-packages/supervisor/supervisord.py|runforever|218] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|handle_read_event|281] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|record_output|215] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|_log|184] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|info|327] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|log|345] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|emit|227] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|doRollover|264])
container_instance  | 2022-02-13 18:45:05,470 INFO exited: oauth_check (exit status 1; not expected)
container_instance  | 2022-02-13 18:45:05,470 INFO exited: oauth_check (exit status 1; not expected)
container_instance  | 2022-02-13 18:45:06,470 INFO gave up: oauth_check entered FATAL state, too many start retries too quickly
container_instance  | 2022-02-13 18:45:06,470 INFO gave up: oauth_check entered FATAL state, too many start retries too quickly

Ran into the same problem. Not sure about the root cause, but as a workaround I set the missing environment variable to the unix socket supervisor is configured to listen to.

I.e. add SUPERVISOR_SERVER_URL=unix://local_path, where local_path is the location of the unix socket specified in /etc/supervisord.conf like this:

[unix_http_server]
file=/var/run/supervisor/supervisor.sock   ; (the path to the socket file)

Edit: default URL seems to be http://localhost:9001

@tyler71 @alxgrk How do I reproduce this? What supervisord version you're using? Ideally, if there's a docker image that produces the error, can you send me a link and I will try it.