Ruler: Reduce delay in syncing rules

[stevesg]

Is your feature request related to a problem? Please describe.

Grafana uses a combination of /ruler/api/v1/rules and /prometheus/api/v1/rules to show rule status. We would like to drop using /ruler/api/v1/rules because it can be very slow in the presence of many rule groups.

However that means we would like /prometheus/api/v1/rules to return a more slightly more consistent view of the configured rules. The syncing doesn't have to be immediate, a handful of seconds would be fine.

Describe the solution you'd like

The existing "Sync" mechanism in the ruler, which tells replicas to reload rule configurations is almost the whole solution. I'm proposing two tweaks to the rule syncing:

• Allow configuration of the sync queue polling intervals
• Only syncing groups known to have been changed (extending the sync queues to be e.g. map[tenant]map[group]).