squad
squad copied to clipboard
Add number of messages in queues in system status page
This will be useful for monitoring squad's health.
Here are the steps for doing so:
- RabbitMQ's host:
- Open ports 15672 and 15671 only for squad-frontend
- Enable management plugin
sudo rabbitmq-plugins enable rabbitmq_management
- Squad-frontent
- GET rabbitmq's host
c = requests.get('http://<private-ip>:15672/api/queues/%2f/ci_fetch/', auth = HTTPBasicAuth('guest', 'guest'))
- Get number of message (check if keys exist, they might not if queue is empty):
c.json()['message_stats']['publish']
- GET rabbitmq's host
There's probably an endpoint to list all the queues, so we don't hard-code queues.
Then add a grafana dash grabbing that endpoint and create an alert if queues grows too much (probably due to worker outage)
- ci_fetch shouldn't have more than 10k tasks
- celery shouldn't have more than 300k tasks