pymongo.errors.ServerSelectionTimeoutError: No servers found yet when trying to start the docker container

[a-mattar]

I created a dataset and started annotating it, and I kept saving my progress. My machine had to restart so the docker container got shutdown. When I try to restart the annotator tool locally using docker-compose up I keep getting the below error and it never starts. What should I do? And is the annotation I worked on before the machine restart lost?

[2022-01-26 11:10:01 +0000] [12] [ERROR] Exception in worker process
[32mannotator_webclient | [0m Traceback (most recent call last):
[32mannotator_webclient | [0m   File "/usr/local/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
[32mannotator_webclient | [0m     worker.init_process()
[32mannotator_webclient | [0m   File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 102, in init_process
[32mannotator_webclient | [0m     super(EventletWorker, self).init_process()
[32mannotator_webclient | [0m   File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
[32mannotator_webclient | [0m     self.load_wsgi()
[32mannotator_webclient | [0m   File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
[32mannotator_webclient | [0m     self.wsgi = self.app.wsgi()
[32mannotator_webclient | [0m   File "/usr/local/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
[32mannotator_webclient | [0m     self.callable = self.load()
[32mannotator_webclient | [0m   File "/usr/local/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
[32mannotator_webclient | [0m     return self.load_wsgiapp()
[32mannotator_webclient | [0m   File "/usr/local/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
[32mannotator_webclient | [0m     return util.import_app(self.app_uri)
[32mannotator_webclient | [0m   File "/usr/local/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
[32mannotator_webclient | [0m     __import__(module)
[32mannotator_webclient | [0m   File "/workspace/webserver/__init__.py", line 63, in <module>
[32mannotator_webclient | [0m     app = create_app()
[32mannotator_webclient | [0m   File "/workspace/webserver/__init__.py", line 57, in create_app
[32mannotator_webclient | [0m     ImageModel.objects.update(annotating=[])
[32mannotator_webclient | [0m   File "/usr/local/lib/python3.6/site-packages/mongoengine/queryset/manager.py", line 37, in __get__
[32mannotator_webclient | [0m     queryset = queryset_class(owner, owner._get_collection())
[32mannotator_webclient | [0m   File "/usr/local/lib/python3.6/site-packages/mongoengine/document.py", line 208, in _get_collection
[32mannotator_webclient | [0m     db.client.is_primary:
[32mannotator_webclient | [0m   File "/usr/local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 857, in is_primary
[32mannotator_webclient | [0m     return self._server_property('is_writable')
[32mannotator_webclient | [0m   File "/usr/local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 689, in _server_property
[32mannotator_webclient | [0m     writable_server_selector)
[32mannotator_webclient | [0m   File "/usr/local/lib/python3.6/site-packages/pymongo/topology.py", line 224, in select_server
[32mannotator_webclient | [0m     address))
[32mannotator_webclient | [0m   File "/usr/local/lib/python3.6/site-packages/pymongo/topology.py", line 183, in select_servers
[32mannotator_webclient | [0m     selector, server_timeout, address)
[32mannotator_webclient | [0m   File "/usr/local/lib/python3.6/site-packages/pymongo/topology.py", line 199, in _select_servers_loop
[32mannotator_webclient | [0m     self._error_message(selector))
[32mannotator_webclient | [0m pymongo.errors.ServerSelectionTimeoutError: No servers found yet
[32mannotator_webclient | [0m [2022-01-26 11:10:01 +0000] [12] [INFO] Worker exiting (pid: 12)
[32mannotator_webclient | [0m [2022-01-26 11:10:03 +0000] [8] [INFO] Shutting down: Master
[32mannotator_webclient | [0m [2022-01-26 11:10:03 +0000] [8] [INFO] Reason: Worker failed to boot.
[34mannotator_webclient | [0m [2022-01-26 11:10:06 +0000] [8] [DEBUG] Current configuration:
[34mannotator_webclient | [0m   config: webserver/gunicorn_config.py
[34mannotator_webclient | [0m   bind: ['0.0.0.0:5000']
[34mannotator_webclient | [0m   backlog: 2048
[34mannotator_webclient | [0m   workers: 1
[34mannotator_webclient | [0m   worker_class: eventlet
[34mannotator_webclient | [0m   threads: 1
[34mannotator_webclient | [0m   worker_connections: 1000
[34mannotator_webclient | [0m   max_requests: 0
[34mannotator_webclient | [0m   max_requests_jitter: 0
[34mannotator_webclient | [0m   timeout: 180
[34mannotator_webclient | [0m   graceful_timeout: 30
[34mannotator_webclient | [0m   keepalive: 2
[34mannotator_webclient | [0m   limit_request_line: 4094
[34mannotator_webclient | [0m   limit_request_fields: 100
[34mannotator_webclient | [0m   limit_request_field_size: 8190
[34mannotator_webclient | [0m   reload: False
[34mannotator_webclient | [0m   reload_engine: auto
[34mannotator_webclient | [0m   reload_extra_files: []
[34mannotator_webclient | [0m   spew: False
[34mannotator_webclient | [0m   check_config: False
[34mannotator_webclient | [0m   preload_app: False
[34mannotator_webclient | [0m   sendfile: False
[34mannotator_webclient | [0m   reuse_port: False
[34mannotator_webclient | [0m   chdir: /workspace
[34mannotator_webclient | [0m   daemon: False
[34mannotator_webclient | [0m   raw_env: []
[34mannotator_webclient | [0m   pidfile: None
[34mannotator_webclient | [0m   worker_tmp_dir: None
[34mannotator_webclient | [0m   user: 0
[34mannotator_webclient | [0m   group: 0
[34mannotator_webclient | [0m   umask: 0
[34mannotator_webclient | [0m   initgroups: False
[34mannotator_webclient | [0m   tmp_upload_dir: None
[34mannotator_webclient | [0m   secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
[34mannotator_webclient | [0m   forwarded_allow_ips: ['127.0.0.1']
[34mannotator_webclient | [0m   accesslog: None
[34mannotator_webclient | [0m   disable_redirect_access_to_syslog: False
[34mannotator_webclient | [0m   access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
[34mannotator_webclient | [0m   errorlog: -
[34mannotator_webclient | [0m   loglevel: debug
[34mannotator_webclient | [0m   capture_output: False
[34mannotator_webclient | [0m   logger_class: gunicorn.glogging.Logger
[34mannotator_webclient | [0m   logconfig: None
[34mannotator_webclient | [0m   logconfig_dict: {}
[34mannotator_webclient | [0m   syslog_addr: udp://localhost:514
[34mannotator_webclient | [0m   syslog: False
[34mannotator_webclient | [0m   syslog_prefix: None
[34mannotator_webclient | [0m   syslog_facility: user
[34mannotator_webclient | [0m   enable_stdio_inheritance: False
[34mannotator_webclient | [0m   statsd_host: None
[34mannotator_webclient | [0m   statsd_prefix: 
[34mannotator_webclient | [0m   proc_name: None
[34mannotator_webclient | [0m   default_proc_name: webserver:app
[34mannotator_webclient | [0m   pythonpath: None
[34mannotator_webclient | [0m   paste: None
[34mannotator_webclient | [0m   on_starting: <function OnStarting.on_starting at 0x7f491acfa2f0>
[34mannotator_webclient | [0m   on_reload: <function OnReload.on_reload at 0x7f491acfa400>
[34mannotator_webclient | [0m   when_ready: <function WhenReady.when_ready at 0x7f491acfa510>
[34mannotator_webclient | [0m   pre_fork: <function Prefork.pre_fork at 0x7f491acfa620>
[34mannotator_webclient | [0m   post_fork: <function Postfork.post_fork at 0x7f491acfa730>
[34mannotator_webclient | [0m   post_worker_init: <function PostWorkerInit.post_worker_init at 0x7f491acfa840>
[34mannotator_webclient | [0m   worker_int: <function WorkerInt.worker_int at 0x7f491acfa950>
[34mannotator_webclient | [0m   worker_abort: <function WorkerAbort.worker_abort at 0x7f491acfaa60>
[34mannotator_webclient | [0m   pre_exec: <function PreExec.pre_exec at 0x7f491acfab70>
[34mannotator_webclient | [0m   pre_request: <function PreRequest.pre_request at 0x7f491acfac80>
[34mannotator_webclient | [0m   post_request: <function PostRequest.post_request at 0x7f491acfad08>
[34mannotator_webclient | [0m   child_exit: <function ChildExit.child_exit at 0x7f491acfae18>
[34mannotator_webclient | [0m   worker_exit: <function WorkerExit.worker_exit at 0x7f491acfaf28>
[34mannotator_webclient | [0m   nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7f491aa760d0>
[34mannotator_webclient | [0m   on_exit: <function OnExit.on_exit at 0x7f491aa761e0>
[34mannotator_webclient | [0m   proxy_protocol: False
[34mannotator_webclient | [0m   proxy_allow_ips: ['127.0.0.1']
[34mannotator_webclient | [0m   keyfile: None
[34mannotator_webclient | [0m   certfile: None
[34mannotator_webclient | [0m   ssl_version: 2
[34mannotator_webclient | [0m   cert_reqs: 0
[34mannotator_webclient | [0m   ca_certs: None
[34mannotator_webclient | [0m   suppress_ragged_eofs: True
[34mannotator_webclient | [0m   do_handshake_on_connect: False
[34mannotator_webclient | [0m   ciphers: TLSv1
[34mannotator_webclient | [0m   raw_paste_global_conf: []

[ryota2425-murakami]

What could be the cause of this? I have the same problem.

[akiross]

This happened to me as well: due to time constraints I couldn't find out the root cause, but in my case the problem was due to failure in resolving the database hostname inside a gunicorn worker when running in docker.

Meaning: when running a test code such as the following inside the coco-annotator docker compose:

from pymongo import MongoClient
print(MongoClient(host='database', port=27017).list_database_names())

that would work from a python interpreter, but not from inside a gunicorn worker.

The workaround I used, although not great, was to resolve the IP of the database before starting gunicorn and saving it in the MONGODB_HOST environment variable. Inside the docker-compose.yml, I used

services:
  webserver:
    command: sh -c "MONGODB_HOST=$$(getent hosts database | head -n1 | awk '{ print $$1}') gunicorn -c webserver/gunicorn_config.py webserver:app --no-sendfile --timeout 180"
  # etc

[EddyJensR]

@akiross your solution works (all containers are deployed), but then .thumbnails files creation fails on a Linux server (shown in image). Without this command, the coco-annotator project in local in macos ventura 13.4.1 it works perfectly!