django-prometheus
django-prometheus copied to clipboard
Unable to start Django shell, OSError: [Errno 98] Address in use
I can't start a Django shell on a host which is currently running my Django project because python manage.py shell
tries to start the metrics endpoint on port 8001.
Traceback (most recent call last):
File "manage.py", line 15, in <module>
execute_from_command_line(sys.argv)
File "/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
utility.execute()
File "/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 347, in execute
django.setup()
File "/venv/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/venv/lib/python3.6/site-packages/django/apps/registry.py", line 120, in populate
app_config.ready()
File "/venv/lib/python3.6/site-packages/django_prometheus/apps.py", line 22, in ready
SetupPrometheusExportsFromConfig()
File "/venv/lib/python3.6/site-packages/django_prometheus/exports.py", line 102, in SetupPrometheusExportsFromConfig
SetupPrometheusEndpointOnPort(port, addr)
File "/venv/lib/python3.6/site-packages/django_prometheus/exports.py", line 44, in SetupPrometheusEndpointOnPort
prometheus_client.start_http_server(port, addr=addr)
File "/venv/lib/python3.6/site-packages/prometheus_client/exposition.py", line 129, in start_http_server
httpd = _ThreadingSimpleServer((addr, port), CustomMetricsHandler)
File "/usr/local/lib/python3.6/socketserver.py", line 453, in __init__
self.server_bind()
File "/usr/local/lib/python3.6/http/server.py", line 136, in server_bind
socketserver.TCPServer.server_bind(self)
File "/usr/local/lib/python3.6/socketserver.py", line 467, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address in use
Prometheus thread should not be started in a Django shell environment.
This has been reported previously, django_prometheus also starts when running other management commands like migrate
. I don't have a good solution, except maybe using the URL router instead of serving on a distinct port, but that's an all or nothing solution.
Suggestions welcome.
Hi - is there a known workaround? I make heavy use of management commands, and sometimes >1 are running at a time. It seems I can't use django-prometheus then? :(
You can set an environment variable when you're in a Django worker process. For instance with Gunicorn you can use the post_fork
hook to, say, set IS_GUNICORN_WORKER
environment variable, then in your Django settings you can set PROMETHEUS_METRICS_EXPORT_PORT[_RANGE]
only if this variable is set.
As a workaround, you can do this in your base settings.py
if config("PROMETHEUS_ENABLE", cast=bool, default=False):
# there was a bug here. when we import libs and then assign env variables, the metrics didnt store
PROMETHEUS_MULTIPROC_DIR = config("PROMETHEUS_MULTIPROC_DIR")
if not os.environ.get("PROMETHEUS_MULTIPROC_DIR"):
os.environ["PROMETHEUS_MULTIPROC_DIR"] = config("PROMETHEUS_MULTIPROC_DIR")
if not os.path.exists(PROMETHEUS_MULTIPROC_DIR):
os.makedirs(PROMETHEUS_MULTIPROC_DIR)
from prometheus_client.multiprocess import MultiProcessCollector
from prometheus_client import CollectorRegistry, start_http_server
INSTALLED_APPS += ["django_prometheus"]
MIDDLEWARE.insert(0, "django_prometheus.middleware.PrometheusBeforeMiddleware")
MIDDLEWARE.append("django_prometheus.middleware.PrometheusAfterMiddleware")
registry = CollectorRegistry()
MultiProcessCollector(registry)
try:
start_http_server(8001, registry=registry)
except OSError:
"""
first process serves metrics on port 8001, other processes raise error: port already in use
one processes collect metrics from PROMETHEUS_MULTIPROC_DIR
"""
and don't use PROMETHEUS_METRICS_EXPORT_PORT[_RANGE]
and PROMETHEUS_METRICS_EXPORT_ADDRESS
When you serve django with gunicorn, it uses multi processes to serve your application, each process runs django_prometheus/exports.py
prometheus_client.start_http_server(port, addr=addr)
so, we can't use this approach