koios-artifacts
koios-artifacts copied to clipboard
Add data-test.sh
Description
Adds data-tests integration for HAproxy to run advanced test for servers in topology of monitoring nodes. First determines a healthy node to use for comparison from a list of trusted nodes, then tests each server in the topology against it.
If the tested node does not pass tests it is marked for maintenance (MAINT) until next run when it could pass. If a MAINT node passes the data tests, it is brought back to READY state.
To-do:
- [ ] Add cron job setup for running these tests automatically (1 hour)
Where should the reviewer start?
Motivation and context
Which issue it fixes?
How has this been tested?
One thing I am considering is to save the responses from ${HEALTHY_INSTANCE} to use them for comparison instead of re-querying for each server in test topology. Given a lot of servers though, the testing could take a while and responses could change over the course of the script runtime.
TO-DO: Make sure tests check offline data (asset registry cache / pool_offline_data).