coco-annotator
coco-annotator copied to clipboard
Published
20 hours ago •
jsbroks
jsbroks
coco-annotator docker container loops while booting up ==> [ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: timed out.
Dear @jsbroks,
I am following the tutorial you shared in YouTube about installing & running coco-annotator using Docker. I am running in Ubuntu and I installed a fresh version of docker and docker compose.
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.6 LTS
Release: 20.04
Codename: focal
$ docker -v
Docker version 24.0.4, build 3713ee1
$ docker-compose -v
Docker Compose version v2.17.2
This is what I got when I attempt to start coco-annotator:
valerio@sco3002:~/coco-annotator (master)$ docker-compose up
[+] Running 44/44
✔ database 12 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 57.1s
✔ 58690f9b18fc Pull complete 44.5s
✔ b51569e7c507 Pull complete 44.5s
✔ da8ef40b9eca Pull complete 44.5s
✔ fb15d46c38dc Pull complete 44.5s
✔ a0dc15b16822 Pull complete 44.5s
✔ b7a3e92f19af Pull complete 44.6s
✔ ed4a7b863fa1 Pull complete 44.6s
✔ a58b030ea8e4 Pull complete 44.6s
✔ 6aa1ba699846 Pull complete 44.6s
✔ ebc52c729dca Pull complete 44.7s
✔ 52e8c440d4d6 Pull complete 55.2s
✔ 22b97876323d Pull complete 55.2s
✔ workers 15 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 52.5s
✔ e79bb959ec00 Pull complete 5.1s
✔ d4b7902036fe Pull complete 5.3s
✔ 1b2a72d4e030 Pull complete 5.4s
✔ d54db43011fd Pull complete 7.7s
✔ 69d473365bb3 Pull complete 35.1s
✔ 35b4aebab92b Pull complete 35.2s
✔ 60ddbccc0359 Pull complete 35.6s
✔ 0b3c337d0c3e Pull complete 35.6s
✔ 2b75f9fe405b Pull complete 35.7s
✔ 8afafd04385f Pull complete 35.7s
✔ c2da40f3dd21 Pull complete 35.7s
✔ 61f4d37d1625 Pull complete 43.7s
✔ 557b33a4cdcb Pull complete 50.7s
✔ 8e4e2338390e Pull complete 50.9s
✔ af9511f0c320 Pull complete 50.9s
✔ messageq 10 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 54.2s
✔ 9d19ee268e0d Pull complete 48.1s
✔ 682a058fc094 Pull complete 48.1s
✔ c6f199470454 Pull complete 48.1s
✔ e588fdd2bc14 Pull complete 51.5s
✔ b2df8cdb3812 Pull complete 51.5s
✔ eec8ffe01a78 Pull complete 52.1s
✔ ef7e4d422ae7 Pull complete 52.1s
✔ a0337bb83298 Pull complete 52.1s
✔ 79a8a5649e7c Pull complete 52.1s
✔ 15c297f005d6 Pull complete 52.3s
✔ webserver 3 layers [⣿⣿⣿] 0B/0B Pulled 52.5s
✔ b7da3f88928c Pull complete 50.9s
✔ 7adcd72abda2 Pull complete 50.9s
✔ f45d8c5057fd Pull complete 50.9s
[+] Running 6/4
✔ Network coco-annotator_cocoannotator Created 0.0s
✔ Volume "coco-annotator_mongodb_data" Created 0.0s
✔ Container annotator_mongodb Created 0.2s
✔ Container annotator_message_q Created 0.2s
✔ Container annotator_workers Created 0.0s
✔ Container annotator_webclient Created 0.0s
Attaching to annotator_message_q, annotator_mongodb, annotator_webclient, annotator_workers
annotator_workers | fatal: Not a git repository (or any of the parent directories): .git
annotator_webclient | [2023-07-17 08:29:26 +0000] [7] [DEBUG] Current configuration:
annotator_webclient | config: webserver/gunicorn_config.py
annotator_webclient | bind: ['0.0.0.0:5000']
annotator_webclient | backlog: 2048
annotator_webclient | workers: 1
annotator_webclient | worker_class: eventlet
annotator_webclient | threads: 1
annotator_webclient | worker_connections: 1000
annotator_webclient | max_requests: 0
annotator_webclient | max_requests_jitter: 0
annotator_webclient | timeout: 180
annotator_webclient | graceful_timeout: 30
annotator_webclient | keepalive: 2
annotator_webclient | limit_request_line: 4094
annotator_webclient | limit_request_fields: 100
annotator_webclient | limit_request_field_size: 8190
annotator_webclient | reload: False
annotator_webclient | reload_engine: auto
annotator_webclient | reload_extra_files: []
annotator_webclient | spew: False
annotator_webclient | check_config: False
annotator_webclient | preload_app: False
annotator_webclient | sendfile: False
annotator_webclient | reuse_port: False
annotator_webclient | chdir: /workspace
annotator_webclient | daemon: False
annotator_webclient | raw_env: []
annotator_webclient | pidfile: None
annotator_webclient | worker_tmp_dir: None
annotator_webclient | user: 0
annotator_webclient | group: 0
annotator_webclient | umask: 0
annotator_webclient | initgroups: False
annotator_webclient | tmp_upload_dir: None
annotator_webclient | secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
annotator_webclient | forwarded_allow_ips: ['127.0.0.1']
annotator_webclient | accesslog: None
annotator_webclient | disable_redirect_access_to_syslog: False
annotator_webclient | access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
annotator_webclient | errorlog: -
annotator_webclient | loglevel: debug
annotator_webclient | capture_output: False
annotator_webclient | logger_class: gunicorn.glogging.Logger
annotator_webclient | logconfig: None
annotator_webclient | logconfig_dict: {}
annotator_webclient | syslog_addr: udp://localhost:514
annotator_webclient | syslog: False
annotator_webclient | syslog_prefix: None
annotator_webclient | syslog_facility: user
annotator_webclient | enable_stdio_inheritance: False
annotator_webclient | statsd_host: None
annotator_webclient | statsd_prefix:
annotator_webclient | proc_name: None
annotator_webclient | default_proc_name: webserver:app
annotator_webclient | pythonpath: None
annotator_webclient | paste: None
annotator_webclient | on_starting: <function OnStarting.on_starting at 0x7fab207582f0>
annotator_webclient | on_reload: <function OnReload.on_reload at 0x7fab20758400>
annotator_webclient | when_ready: <function WhenReady.when_ready at 0x7fab20758510>
annotator_webclient | pre_fork: <function Prefork.pre_fork at 0x7fab20758620>
annotator_webclient | post_fork: <function Postfork.post_fork at 0x7fab20758730>
annotator_webclient | post_worker_init: <function PostWorkerInit.post_worker_init at 0x7fab20758840>
annotator_webclient | worker_int: <function WorkerInt.worker_int at 0x7fab20758950>
annotator_webclient | worker_abort: <function WorkerAbort.worker_abort at 0x7fab20758a60>
annotator_webclient | pre_exec: <function PreExec.pre_exec at 0x7fab20758b70>
annotator_webclient | pre_request: <function PreRequest.pre_request at 0x7fab20758c80>
annotator_webclient | post_request: <function PostRequest.post_request at 0x7fab20758d08>
annotator_webclient | child_exit: <function ChildExit.child_exit at 0x7fab20758e18>
annotator_webclient | worker_exit: <function WorkerExit.worker_exit at 0x7fab20758f28>
annotator_webclient | nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7fab204d40d0>
annotator_webclient | on_exit: <function OnExit.on_exit at 0x7fab204d41e0>
annotator_webclient | proxy_protocol: False
annotator_webclient | proxy_allow_ips: ['127.0.0.1']
annotator_webclient | keyfile: None
annotator_webclient | certfile: None
annotator_webclient | ssl_version: 2
annotator_webclient | cert_reqs: 0
annotator_webclient | ca_certs: None
annotator_webclient | suppress_ragged_eofs: True
annotator_webclient | do_handshake_on_connect: False
annotator_webclient | ciphers: TLSv1
annotator_webclient | raw_paste_global_conf: []
annotator_webclient | [2023-07-17 08:29:26 +0000] [7] [INFO] Starting gunicorn 19.9.0
annotator_webclient | [2023-07-17 08:29:26 +0000] [7] [DEBUG] Arbiter booted
annotator_webclient | [2023-07-17 08:29:26 +0000] [7] [INFO] Listening at: http://0.0.0.0:5000 (7)
annotator_webclient | [2023-07-17 08:29:26 +0000] [7] [INFO] Using worker: eventlet
annotator_webclient | [2023-07-17 08:29:26 +0000] [11] [INFO] Booting worker with pid: 11
annotator_webclient | [2023-07-17 08:29:26 +0000] [7] [DEBUG] 1 workers
annotator_webclient | [2023-07-17 08:29:27 +0000] [11] [WARNING] MaskRCNN model is disabled.
annotator_webclient | [2023-07-17 08:29:27 +0000] [11] [WARNING] DEXTR model is disabled.
annotator_message_q | 2023-07-17 08:29:28.378703+00:00 [notice] <0.44.0> Application syslog exited with reason: stopped
annotator_message_q | 2023-07-17 08:29:28.380955+00:00 [notice] <0.230.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
annotator_message_q | 2023-07-17 08:29:28.381310+00:00 [notice] <0.230.0> Logging: configured log handlers are now ACTIVE
annotator_message_q | 2023-07-17 08:29:28.479181+00:00 [info] <0.230.0> ra: starting system quorum_queues
annotator_message_q | 2023-07-17 08:29:28.479253+00:00 [info] <0.230.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@29539ca56422/quorum/rabbit@29539ca56422
annotator_message_q | 2023-07-17 08:29:28.505732+00:00 [info] <0.261.0> ra system 'quorum_queues' running pre init for 0 registered servers
annotator_message_q | 2023-07-17 08:29:28.510774+00:00 [info] <0.262.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
annotator_message_q | 2023-07-17 08:29:28.516680+00:00 [notice] <0.267.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables
annotator_message_q | 2023-07-17 08:29:28.519773+00:00 [info] <0.230.0> ra: starting system coordination
annotator_message_q | 2023-07-17 08:29:28.519812+00:00 [info] <0.230.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@29539ca56422/coordination/rabbit@29539ca56422
annotator_message_q | 2023-07-17 08:29:28.520359+00:00 [info] <0.274.0> ra system 'coordination' running pre init for 0 registered servers
annotator_message_q | 2023-07-17 08:29:28.520681+00:00 [info] <0.275.0> ra: meta data store initialised for system coordination. 0 record(s) recovered
annotator_message_q | 2023-07-17 08:29:28.520797+00:00 [notice] <0.280.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables
annotator_message_q | 2023-07-17 08:29:28.522025+00:00 [info] <0.230.0>
annotator_message_q | 2023-07-17 08:29:28.522025+00:00 [info] <0.230.0> Starting RabbitMQ 3.12.1 on Erlang 25.3.2.3 [jit]
annotator_message_q | 2023-07-17 08:29:28.522025+00:00 [info] <0.230.0> Copyright (c) 2007-2023 VMware, Inc. or its affiliates.
annotator_message_q | 2023-07-17 08:29:28.522025+00:00 [info] <0.230.0> Licensed under the MPL 2.0. Website: https://rabbitmq.com
annotator_message_q |
annotator_message_q | ## ## RabbitMQ 3.12.1
annotator_message_q | ## ##
annotator_message_q | ########## Copyright (c) 2007-2023 VMware, Inc. or its affiliates.
annotator_message_q | ###### ##
annotator_message_q | ########## Licensed under the MPL 2.0. Website: https://rabbitmq.com
annotator_message_q |
annotator_message_q | Erlang: 25.3.2.3 [jit]
annotator_message_q | TLS Library: OpenSSL - OpenSSL 3.1.1 30 May 2023
annotator_message_q | Release series support status: supported
annotator_message_q |
annotator_message_q | Doc guides: https://rabbitmq.com/documentation.html
annotator_message_q | Support: https://rabbitmq.com/contact.html
annotator_message_q | Tutorials: https://rabbitmq.com/getstarted.html
annotator_message_q | Monitoring: https://rabbitmq.com/monitoring.html
annotator_message_q |
annotator_message_q | Logs: <stdout>
annotator_message_q |
annotator_message_q | Config file(s): /etc/rabbitmq/conf.d/10-defaults.conf
annotator_message_q | /etc/rabbitmq/conf.d/20-management_agent.disable_metrics_collector.conf
annotator_message_q |
annotator_message_q | Starting broker...2023-07-17 08:29:28.526716+00:00 [info] <0.230.0>
annotator_message_q | 2023-07-17 08:29:28.526716+00:00 [info] <0.230.0> node : rabbit@29539ca56422
annotator_message_q | 2023-07-17 08:29:28.526716+00:00 [info] <0.230.0> home dir : /var/lib/rabbitmq
annotator_message_q | 2023-07-17 08:29:28.526716+00:00 [info] <0.230.0> config file(s) : /etc/rabbitmq/conf.d/10-defaults.conf
annotator_message_q | 2023-07-17 08:29:28.526716+00:00 [info] <0.230.0> : /etc/rabbitmq/conf.d/20-management_agent.disable_metrics_collector.conf
annotator_message_q | 2023-07-17 08:29:28.526716+00:00 [info] <0.230.0> cookie hash : xyJUhoBkOPNciTnQJtbbwQ==
annotator_message_q | 2023-07-17 08:29:28.526716+00:00 [info] <0.230.0> log(s) : <stdout>
annotator_message_q | 2023-07-17 08:29:28.526716+00:00 [info] <0.230.0> data dir : /var/lib/rabbitmq/mnesia/rabbit@29539ca56422
annotator_message_q | 2023-07-17 08:29:29.757980+00:00 [info] <0.230.0> Running boot step pre_boot defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.758034+00:00 [info] <0.230.0> Running boot step rabbit_global_counters defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.758219+00:00 [info] <0.230.0> Running boot step rabbit_osiris_metrics defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.758317+00:00 [info] <0.230.0> Running boot step rabbit_core_metrics defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.758838+00:00 [info] <0.230.0> Running boot step rabbit_alarm defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.760850+00:00 [info] <0.299.0> Memory high watermark set to 12752 MiB (13371775385 bytes) of 31880 MiB (33429438464 bytes) total
annotator_message_q | 2023-07-17 08:29:29.762249+00:00 [info] <0.301.0> Enabling free disk space monitoring (disk free space: 133362720768, total memory: 33429438464)
annotator_message_q | 2023-07-17 08:29:29.762303+00:00 [info] <0.301.0> Disk free limit set to 50MB
annotator_message_q | 2023-07-17 08:29:29.763119+00:00 [info] <0.230.0> Running boot step code_server_cache defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.763181+00:00 [info] <0.230.0> Running boot step file_handle_cache defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.763313+00:00 [info] <0.304.0> Limiting to approx 1048479 file handles (943629 sockets)
annotator_message_q | 2023-07-17 08:29:29.763395+00:00 [info] <0.305.0> FHC read buffering: OFF
annotator_message_q | 2023-07-17 08:29:29.763435+00:00 [info] <0.305.0> FHC write buffering: ON
annotator_message_q | 2023-07-17 08:29:29.763614+00:00 [info] <0.230.0> Running boot step worker_pool defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.763659+00:00 [info] <0.282.0> Will use 12 processes for default worker pool
annotator_message_q | 2023-07-17 08:29:29.763694+00:00 [info] <0.282.0> Starting worker pool 'worker_pool' with 12 processes in it
annotator_message_q | 2023-07-17 08:29:29.764046+00:00 [info] <0.230.0> Running boot step database defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.764299+00:00 [info] <0.230.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@29539ca56422 is empty. Assuming we need to join an existing cluster or initialise from scratch...
annotator_message_q | 2023-07-17 08:29:29.764339+00:00 [info] <0.230.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
annotator_message_q | 2023-07-17 08:29:29.764365+00:00 [info] <0.230.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
annotator_message_q | 2023-07-17 08:29:29.764416+00:00 [info] <0.230.0> All discovered existing cluster peers:
annotator_message_q | 2023-07-17 08:29:29.764438+00:00 [info] <0.230.0> Discovered no peer nodes to cluster with. Some discovery backends can filter nodes out based on a readiness criteria. Enabling debug logging might help troubleshoot.
annotator_message_q | 2023-07-17 08:29:29.766045+00:00 [notice] <0.44.0> Application mnesia exited with reason: stopped
annotator_message_q | 2023-07-17 08:29:29.807749+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
annotator_message_q | 2023-07-17 08:29:29.807842+00:00 [info] <0.230.0> Successfully synced tables from a peer
annotator_message_q | 2023-07-17 08:29:29.808164+00:00 [notice] <0.283.0> Feature flags: attempt to enable `stream_sac_coordinator_unblock_group`...
annotator_message_q | 2023-07-17 08:29:29.829500+00:00 [notice] <0.283.0> Feature flags: `stream_sac_coordinator_unblock_group` enabled
annotator_message_q | 2023-07-17 08:29:29.829645+00:00 [notice] <0.283.0> Feature flags: attempt to enable `restart_streams`...
annotator_message_q | 2023-07-17 08:29:29.851801+00:00 [notice] <0.283.0> Feature flags: `restart_streams` enabled
annotator_message_q | 2023-07-17 08:29:29.851965+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
annotator_message_q | 2023-07-17 08:29:29.852057+00:00 [info] <0.230.0> Successfully synced tables from a peer
annotator_message_q | 2023-07-17 08:29:29.861128+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
annotator_message_q | 2023-07-17 08:29:29.861263+00:00 [info] <0.230.0> Successfully synced tables from a peer
annotator_message_q | 2023-07-17 08:29:29.861308+00:00 [info] <0.230.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
annotator_message_q | 2023-07-17 08:29:29.861348+00:00 [info] <0.230.0> Will try to unlock with peer discovery backend rabbit_peer_discovery_classic_config
annotator_message_q | 2023-07-17 08:29:29.861494+00:00 [info] <0.230.0> Running boot step tracking_metadata_store defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.861572+00:00 [info] <0.490.0> Setting up a table for connection tracking on this node: tracked_connection
annotator_message_q | 2023-07-17 08:29:29.861625+00:00 [info] <0.490.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost
annotator_message_q | 2023-07-17 08:29:29.861681+00:00 [info] <0.490.0> Setting up a table for per-user connection counting on this node: tracked_connection_per_user
annotator_message_q | 2023-07-17 08:29:29.861724+00:00 [info] <0.490.0> Setting up a table for channel tracking on this node: tracked_channel
annotator_message_q | 2023-07-17 08:29:29.861762+00:00 [info] <0.490.0> Setting up a table for channel tracking on this node: tracked_channel_per_user
annotator_message_q | 2023-07-17 08:29:29.861816+00:00 [info] <0.230.0> Running boot step networking_metadata_store defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.861876+00:00 [info] <0.230.0> Running boot step feature_flags defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.861969+00:00 [info] <0.230.0> Running boot step codec_correctness_check defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862015+00:00 [info] <0.230.0> Running boot step external_infrastructure defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862047+00:00 [info] <0.230.0> Running boot step rabbit_event defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862145+00:00 [info] <0.230.0> Running boot step rabbit_registry defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862202+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862262+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862322+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862385+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862425+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862461+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862501+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862543+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862580+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862619+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862658+00:00 [info] <0.230.0> Running boot step rabbit_priority_queue defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862689+00:00 [info] <0.230.0> Priority queues enabled, real BQ is rabbit_variable_queue
annotator_message_q | 2023-07-17 08:29:29.862729+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862769+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862813+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_random defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862882+00:00 [info] <0.230.0> Running boot step kernel_ready defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862907+00:00 [info] <0.230.0> Running boot step rabbit_sysmon_minder defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.862996+00:00 [info] <0.230.0> Running boot step rabbit_epmd_monitor defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.863375+00:00 [info] <0.498.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
annotator_message_q | 2023-07-17 08:29:29.863457+00:00 [info] <0.230.0> Running boot step guid_generator defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.864356+00:00 [info] <0.230.0> Running boot step rabbit_node_monitor defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.864498+00:00 [info] <0.502.0> Starting rabbit_node_monitor
annotator_message_q | 2023-07-17 08:29:29.864568+00:00 [info] <0.230.0> Running boot step delegate_sup defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.864910+00:00 [info] <0.230.0> Running boot step rabbit_memory_monitor defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.865042+00:00 [info] <0.230.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.865100+00:00 [info] <0.230.0> Running boot step core_initialized defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.865126+00:00 [info] <0.230.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.865155+00:00 [info] <0.230.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.865187+00:00 [info] <0.230.0> Running boot step rabbit_definitions_hashing defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.865242+00:00 [info] <0.230.0> Running boot step rabbit_exchange_parameters defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.865451+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.865677+00:00 [info] <0.230.0> Running boot step rabbit_policies defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.865913+00:00 [info] <0.230.0> Running boot step rabbit_policy defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.865980+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_validator defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.866044+00:00 [info] <0.230.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.866085+00:00 [info] <0.230.0> Running boot step rabbit_stream_coordinator defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.866173+00:00 [info] <0.230.0> Running boot step rabbit_vhost_limit defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.866226+00:00 [info] <0.230.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
annotator_message_q | 2023-07-17 08:29:29.866260+00:00 [info] <0.230.0> Management plugin: using rates mode 'basic'
annotator_message_q | 2023-07-17 08:29:29.866419+00:00 [info] <0.230.0> Running boot step recovery defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.866906+00:00 [info] <0.230.0> Running boot step empty_db_check defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.866949+00:00 [info] <0.230.0> Will seed default virtual host and user...
annotator_message_q | 2023-07-17 08:29:29.867010+00:00 [info] <0.230.0> Adding vhost '/' (description: 'Default virtual host', tags: [])
annotator_message_q | 2023-07-17 08:29:29.868513+00:00 [info] <0.230.0> Inserted a virtual host record {vhost,<<"/">>,[],
annotator_message_q | 2023-07-17 08:29:29.868513+00:00 [info] <0.230.0> #{description =>
annotator_message_q | 2023-07-17 08:29:29.868513+00:00 [info] <0.230.0> <<"Default virtual host">>,
annotator_message_q | 2023-07-17 08:29:29.868513+00:00 [info] <0.230.0> tags => []}}
annotator_message_q | 2023-07-17 08:29:29.874012+00:00 [info] <0.547.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@29539ca56422/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
annotator_message_q | 2023-07-17 08:29:29.874500+00:00 [info] <0.547.0> Setting segment_entry_count for vhost '/' with 0 queues to '2048'
annotator_message_q | 2023-07-17 08:29:29.876120+00:00 [info] <0.547.0> Starting message stores for vhost '/'
annotator_message_q | 2023-07-17 08:29:29.876272+00:00 [info] <0.557.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
annotator_message_q | 2023-07-17 08:29:29.876856+00:00 [info] <0.547.0> Started message store of type transient for vhost '/'
annotator_message_q | 2023-07-17 08:29:29.876971+00:00 [info] <0.561.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
annotator_message_q | 2023-07-17 08:29:29.877336+00:00 [warning] <0.561.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
annotator_message_q | 2023-07-17 08:29:29.877804+00:00 [info] <0.547.0> Started message store of type persistent for vhost '/'
annotator_message_q | 2023-07-17 08:29:29.877894+00:00 [info] <0.547.0> Recovering 0 queues of type rabbit_classic_queue took 3ms
annotator_message_q | 2023-07-17 08:29:29.877940+00:00 [info] <0.547.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
annotator_message_q | 2023-07-17 08:29:29.877974+00:00 [info] <0.547.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
annotator_message_q | 2023-07-17 08:29:29.879609+00:00 [info] <0.230.0> Created user 'user'
annotator_message_q | 2023-07-17 08:29:29.880371+00:00 [info] <0.230.0> Successfully set user tags for user 'user' to [administrator]
annotator_message_q | 2023-07-17 08:29:29.881166+00:00 [info] <0.230.0> Successfully set permissions for user 'user' in virtual host '/' to '.*', '.*', '.*'
annotator_message_q | 2023-07-17 08:29:29.881209+00:00 [info] <0.230.0> Running boot step rabbit_observer_cli defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.881265+00:00 [info] <0.230.0> Running boot step rabbit_looking_glass defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.881294+00:00 [info] <0.230.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.881385+00:00 [info] <0.230.0> Running boot step background_gc defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.881487+00:00 [info] <0.230.0> Running boot step routing_ready defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.881525+00:00 [info] <0.230.0> Running boot step pre_flight defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.881549+00:00 [info] <0.230.0> Running boot step notify_cluster defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.881575+00:00 [info] <0.230.0> Running boot step networking defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.881604+00:00 [info] <0.230.0> Running boot step definition_import_worker_pool defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.881644+00:00 [info] <0.282.0> Starting worker pool 'definition_import_pool' with 12 processes in it
annotator_message_q | 2023-07-17 08:29:29.882123+00:00 [info] <0.230.0> Running boot step cluster_name defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.882181+00:00 [info] <0.230.0> Initialising internal cluster ID to 'rabbitmq-cluster-id-eQfBJ6nw978Ey1-d__x7KQ'
annotator_message_q | 2023-07-17 08:29:29.882977+00:00 [info] <0.230.0> Running boot step direct_client defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.883049+00:00 [info] <0.230.0> Running boot step rabbit_maintenance_mode_state defined by app rabbit
annotator_message_q | 2023-07-17 08:29:29.883093+00:00 [info] <0.230.0> Creating table rabbit_node_maintenance_states for maintenance mode status
annotator_message_q | 2023-07-17 08:29:29.884551+00:00 [info] <0.607.0> Resetting node maintenance status
annotator_message_q | 2023-07-17 08:29:29.893547+00:00 [info] <0.634.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
annotator_message_q | 2023-07-17 08:29:29.893684+00:00 [info] <0.607.0> Ready to start client connection listeners
annotator_message_q | 2023-07-17 08:29:29.894988+00:00 [info] <0.678.0> started TCP listener on [::]:5672
annotator_message_q | completed with 3 plugins.
annotator_message_q | 2023-07-17 08:29:29.939302+00:00 [info] <0.607.0> Server startup complete; 3 plugins started.
annotator_message_q | 2023-07-17 08:29:29.939302+00:00 [info] <0.607.0> * rabbitmq_prometheus
annotator_message_q | 2023-07-17 08:29:29.939302+00:00 [info] <0.607.0> * rabbitmq_web_dispatch
annotator_message_q | 2023-07-17 08:29:29.939302+00:00 [info] <0.607.0> * rabbitmq_management_agent
annotator_workers | /usr/local/lib/python3.6/site-packages/celery/platforms.py:796: RuntimeWarning: You're running the worker with superuser privileges: this is
annotator_workers | absolutely not recommended!
annotator_workers |
annotator_workers | Please specify a different user using the --uid option.
annotator_workers |
annotator_workers | User information: uid=0 euid=0 gid=0 egid=0
annotator_workers |
annotator_workers | uid=uid, euid=euid, gid=gid, egid=egid,
annotator_workers | [2023-07-17 08:29:31,570: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: timed out.
annotator_workers | Trying again in 2.00 seconds...
annotator_workers |
annotator_workers | [2023-07-17 08:29:37,580: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: timed out.
annotator_workers | Trying again in 4.00 seconds...
annotator_workers |
annotator_workers | [2023-07-17 08:29:45,598: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: timed out.
annotator_workers | Trying again in 6.00 seconds...
annotator_workers |
annotator_workers | [2023-07-17 08:29:55,622: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: timed out.
annotator_workers | Trying again in 8.00 seconds...
annotator_workers |
annotator_webclient | [2023-07-17 08:29:57 +0000] [11] [ERROR] Exception in worker process
annotator_webclient | Traceback (most recent call last):
annotator_webclient | File "/usr/local/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
annotator_webclient | worker.init_process()
annotator_webclient | File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 102, in init_process
annotator_webclient | super(EventletWorker, self).init_process()
annotator_webclient | File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
annotator_webclient | self.load_wsgi()
annotator_webclient | File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
annotator_webclient | self.wsgi = self.app.wsgi()
annotator_webclient | File "/usr/local/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
annotator_webclient | self.callable = self.load()
annotator_webclient | File "/usr/local/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
annotator_webclient | return self.load_wsgiapp()
annotator_webclient | File "/usr/local/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
annotator_webclient | return util.import_app(self.app_uri)
annotator_webclient | File "/usr/local/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
annotator_webclient | __import__(module)
annotator_webclient | File "/workspace/webserver/__init__.py", line 63, in <module>
annotator_webclient | app = create_app()
annotator_webclient | File "/workspace/webserver/__init__.py", line 57, in create_app
annotator_webclient | ImageModel.objects.update(annotating=[])
annotator_webclient | File "/usr/local/lib/python3.6/site-packages/mongoengine/queryset/manager.py", line 37, in __get__
annotator_webclient | queryset = queryset_class(owner, owner._get_collection())
annotator_webclient | File "/usr/local/lib/python3.6/site-packages/mongoengine/document.py", line 208, in _get_collection
annotator_webclient | db.client.is_primary:
annotator_webclient | File "/usr/local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 857, in is_primary
annotator_webclient | return self._server_property('is_writable')
annotator_webclient | File "/usr/local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 689, in _server_property
annotator_webclient | writable_server_selector)
annotator_webclient | File "/usr/local/lib/python3.6/site-packages/pymongo/topology.py", line 224, in select_server
annotator_webclient | address))
annotator_webclient | File "/usr/local/lib/python3.6/site-packages/pymongo/topology.py", line 183, in select_servers
annotator_webclient | selector, server_timeout, address)
annotator_webclient | File "/usr/local/lib/python3.6/site-packages/pymongo/topology.py", line 199, in _select_servers_loop
annotator_webclient | self._error_message(selector))
annotator_webclient | pymongo.errors.ServerSelectionTimeoutError: database:27017: timed out
annotator_webclient | [2023-07-17 08:29:57 +0000] [11] [INFO] Worker exiting (pid: 11)
annotator_webclient | [2023-07-17 08:29:58 +0000] [7] [INFO] Shutting down: Master
annotator_webclient | [2023-07-17 08:29:58 +0000] [7] [INFO] Reason: Worker failed to boot.
annotator_webclient exited with code 0
I see no logs for annotator_mongodb image and you have an error about mongodb access: annotator_webclient | pymongo.errors.ServerSelectionTimeoutError: database:27017: timed out
So your problem seem's related to mongodb.
I would try to run: docker-compose up annotator_mongodb
docker compose up starup coco-annotartor must be like this (4 container)
- annotator_webclient
- annotator_workers
- annotator_mongodb
- annotator_message_q
