[🐛 Bug]: Provisioned provider not updated with new configurations

[tuantran0910]

Hi team, I found a bug when we provision providers via KEEP_PROVIDERS OR KEEP_PROVIDER_DIRECTORY. If the first time, I provision this provider:

KEEP_PROVIDERS={"keepVictoriaMetrics":{"type":"victoriametrics","authentication":{"VMAlertHost":"http://vm.host.com","VMAlertPort":5678}}

So if I now want to change the configure of the existing provisioned provider (not change its name but let's say change the host name to http://localhost)

KEEP_PROVIDERS={"keepVictoriaMetrics":{"type":"victoriametrics","authentication":{"VMAlertHost":"http://localhost","VMAlertPort":1234}}

But unfortunately, this new configurations will not be updated into existing provisioned provider. This is because of those logic:

if ProvidersService.is_provider_installed(tenant_id, provider_name):
    logger.info(f"Provider {provider_name} already installed")
    continue

Cannot update the existing provisioned provider. So I think there is better to drop this conditional statement check.

However, to make the provider configurations to be updated, so I think we should:

1. First, delete all existing provisioned providers.
2. Then, provision new providers from either the environment variable or the directory.
3. If any error occurs, rollback everything to the original state.

So each time the backend starts, restarts, ... The progress re-provisioning providers will happen. What if each time the backend restarts, the configurations not change, so I think we first calculate the hash of the configurations, then save it in either redis or local file system (in dir state).