[feature request] Make receiver relabelConfig reloadable

[hanjm]

Is your proposal related to a problem?

Thanks #5391 support relabel on receiver router, It is very useful feature to drop high cardinarly metrics to self protect. It is static config, need reboot when config update.

Describe the solution you'd like

Use fsnotify watch file change and reload it.

Additional context

1. Create metric about reload result and timestamp.
2. Use atomic value instead of mutex lock, avoid block receiver ingest metric.

[piclemx]

@hanjm I can try to do that this week if you still need help.

[yeya24]

@piclemx Please go ahead. This feature would be a nice addition.

[stale[bot]]

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[juanrh]

I'd like to work on this

[piclemx]

@juanrh go ahead. I started something, but I drop it because I was changing too many things.

[juanrh]

I have https://github.com/juanrh/thanos/commits/add-path-content-reloader-with-relabeller-2 for this but it depends on https://github.com/thanos-io/thanos/pull/5673 that adds a generic mechanism for reloading configuration. I should be able to have a PR for this soon after that is merged