Error when Prometheus enabled: OSError: [Errno 98] Address already in use

[fabriziocacicia]

I'm getting an error trying to executing the view command if I have Prometheus enabled. This is what I get:

Traceback (most recent call last):
  File "manager.py", line 37, in <module>
    view()
  File "/home/chia/Swar-Chia-Plot-Manager/plotmanager/library/utilities/commands.py", line 162, in view
    jobs, running_work = get_running_plots(jobs=jobs, running_work=running_work,
  File "/home/chia/Swar-Chia-Plot-Manager/plotmanager/library/utilities/processes.py", line 254, in get_running_plots
    set_plots_running(total_running_plots=assumed_job.total_running, job_name=assumed_job.name,
  File "/home/chia/Swar-Chia-Plot-Manager/plotmanager/library/utilities/instrumentation.py", line 24, in set_plots_running
    _get_metrics(instrumentation_settings=instrumentation_settings)
  File "/home/chia/Swar-Chia-Plot-Manager/plotmanager/library/utilities/instrumentation.py", line 19, in _get_metrics
    start_http_server(port)
  File "/home/chia/Swar-Chia-Plot-Manager/swar/lib/python3.8/site-packages/prometheus_client/exposition.py", line 149, in start_wsgi_server
    httpd = make_server(addr, port, app, ThreadingWSGIServer, handler_class=_SilentHandler)
  File "/usr/lib/python3.8/wsgiref/simple_server.py", line 154, in make_server
    server = server_class((host, port), handler_class)
  File "/usr/lib/python3.8/socketserver.py", line 452, in __init__
    self.server_bind()
  File "/usr/lib/python3.8/wsgiref/simple_server.py", line 50, in server_bind
    HTTPServer.server_bind(self)
  File "/usr/lib/python3.8/http/server.py", line 138, in server_bind
    socketserver.TCPServer.server_bind(self)
  File "/usr/lib/python3.8/socketserver.py", line 466, in server_bind
    self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use

[anufriu]

caused by this command start_http_server(port) when you start "view" or "json" or "status" command it triggers in instrumentation.py uses processes.py and triggers set_plots_running(total_running_plots=assumed_job.total_running, job_name=assumed_job.name, instrumentation_settings=instrumentation_settings) at string 255 for no reason.

[anufriu]

the key is http_server prometheus already started and binded at this port.

[fabriziocacicia]

the key is http_server prometheus already started and binded at this port.

You are saying that I should change the port of the Prometheus http server in order to avoid that issue?

[anufriu]

nope. when you restart the plotmanager it will bind the port again) workaround is to start main process then comment start_http_server(port) string and save the file. after that you can start using all the features , but if you restart main plotmanager process prometheus feature stop working too.

[fabriziocacicia]

Thanks, I will try so. Anyway what should be the intended way to use the Prometheus integrations without editing the code?

[anufriu]

we only can wait when dev implement new logic and it will stop trying to set up new http server every time.

[anufriu]

https://github.com/swar/Swar-Chia-Plot-Manager/pull/824/files

[ieguiguren]

An easier way to overcome this issue until it's fixed is to start the manager with prometheus enabled and, once it's started, disabling prometheus in the config. Then you'll be able to start the view, json and so...

[bouncethebox]

An easier way to overcome this issue until it's fixed is to start the manager with prometheus enabled and, once it's started, disabling prometheus in the config. Then you'll be able to start the view, json and so...

Holy fricken' crahp! This works quite well.

[pikolos]

An easier way to overcome this issue until it's fixed is to start the manager with prometheus enabled and, once it's started, disabling prometheus in the config. Then you'll be able to start the view, json and so...

doing this seems to work but all the metrics are no longer updated.