Health Checks

[ghjm]

Use the operator's credentials to connect to the ClickHouse instance and run health checks:

• Access point is available (use chi level service)
• Distributed query check: SELECT count() FROM cluster('all-sharded', cluster('all-sharded', system.one))
• Zookeeper check -- run only if zookeeper is a part of CHI spec.configuration: SELECT count() FROM system.zookeeper WHERE path = '/'
• No readonly replicas: SELECT max(value) FROM cluster('{cluster}', system.metrics) WHERE metric = 'ReadonlyReplica'
• No delayed inserts: SELECT value FROM system.metrics WHERE metric = 'DelayedInserts'
• Healthy schema: MaxPartCountForPartition: >150 yellow, >300 red: select value from system.asynchronous_metrics where metric='MaxPartCountForPartition'

Provide some kind of warning if the health checks are failing.