Unable to start Django shell, OSError: [Errno 98] Address in use

[elnappo]

I can't start a Django shell on a host which is currently running my Django project because python manage.py shell tries to start the metrics endpoint on port 8001.

Traceback (most recent call last):
  File "manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 347, in execute
    django.setup()
  File "/venv/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/venv/lib/python3.6/site-packages/django/apps/registry.py", line 120, in populate
    app_config.ready()
  File "/venv/lib/python3.6/site-packages/django_prometheus/apps.py", line 22, in ready
    SetupPrometheusExportsFromConfig()
  File "/venv/lib/python3.6/site-packages/django_prometheus/exports.py", line 102, in SetupPrometheusExportsFromConfig
    SetupPrometheusEndpointOnPort(port, addr)
  File "/venv/lib/python3.6/site-packages/django_prometheus/exports.py", line 44, in SetupPrometheusEndpointOnPort
    prometheus_client.start_http_server(port, addr=addr)
  File "/venv/lib/python3.6/site-packages/prometheus_client/exposition.py", line 129, in start_http_server
    httpd = _ThreadingSimpleServer((addr, port), CustomMetricsHandler)
  File "/usr/local/lib/python3.6/socketserver.py", line 453, in __init__
    self.server_bind()
  File "/usr/local/lib/python3.6/http/server.py", line 136, in server_bind
    socketserver.TCPServer.server_bind(self)
  File "/usr/local/lib/python3.6/socketserver.py", line 467, in server_bind
    self.socket.bind(self.server_address)
OSError: [Errno 98] Address in use

Prometheus thread should not be started in a Django shell environment.

[korfuri]

This has been reported previously, django_prometheus also starts when running other management commands like migrate. I don't have a good solution, except maybe using the URL router instead of serving on a distinct port, but that's an all or nothing solution.

Suggestions welcome.

[manos]

Hi - is there a known workaround? I make heavy use of management commands, and sometimes >1 are running at a time. It seems I can't use django-prometheus then? :(

[sdanzan]

You can set an environment variable when you're in a Django worker process. For instance with Gunicorn you can use the post_fork hook to, say, set IS_GUNICORN_WORKER environment variable, then in your Django settings you can set PROMETHEUS_METRICS_EXPORT_PORT[_RANGE] only if this variable is set.

[saeedAdelpour]

As a workaround, you can do this in your base settings.py

if config("PROMETHEUS_ENABLE", cast=bool, default=False):
    # there was a bug here. when we import libs and then assign env variables, the metrics didnt store
    
    PROMETHEUS_MULTIPROC_DIR = config("PROMETHEUS_MULTIPROC_DIR")
    if not os.environ.get("PROMETHEUS_MULTIPROC_DIR"):
        os.environ["PROMETHEUS_MULTIPROC_DIR"] = config("PROMETHEUS_MULTIPROC_DIR")
    if not os.path.exists(PROMETHEUS_MULTIPROC_DIR):
        os.makedirs(PROMETHEUS_MULTIPROC_DIR)

    from prometheus_client.multiprocess import MultiProcessCollector
    from prometheus_client import CollectorRegistry, start_http_server
    
    INSTALLED_APPS += ["django_prometheus"]
    MIDDLEWARE.insert(0, "django_prometheus.middleware.PrometheusBeforeMiddleware")
    MIDDLEWARE.append("django_prometheus.middleware.PrometheusAfterMiddleware")
    
    
    registry = CollectorRegistry()
    MultiProcessCollector(registry)
    try:
        start_http_server(8001, registry=registry)
    except OSError:
        """
        first process serves metrics on port 8001, other processes raise error: port already in use
        one processes collect metrics from PROMETHEUS_MULTIPROC_DIR
        """

and don't use PROMETHEUS_METRICS_EXPORT_PORT[_RANGE] and PROMETHEUS_METRICS_EXPORT_ADDRESS

When you serve django with gunicorn, it uses multi processes to serve your application, each process runs django_prometheus/exports.py

prometheus_client.start_http_server(port, addr=addr)

so, we can't use this approach