Allow to select storageClass per zone

[blazarecki]

What this PR does

Add option to configure the storageClass per zone when zoneAwareReplication is enabled.

Why this PR

When I enabled zoneAwareReplication for ingesters or store-gateway (should be the same for alertmanager), I notice that the Persistent Volume was not always created in the good zone (the pod one).

Only 1 out of 3 Persistent Volume was accessible for the pod.

Exemple:

Pod: ingester-zone-a-0, Persistent Volume: zone-a -> OK 
Pod: ingester-zone-a-1, Persistent Volume: zone-b -> KO
Pod: ingester-zone-a-2, Persistent Volume: zone-c -> KO
Pod: ingester-zone-a-3, Persistent Volume: zone-a -> OK

To fix it I create one Storage Class per Zone and update the volumeClaimTemplates spec to use the storageClassName from the rollout.

      zones:
        - name: zone-a
          storageClass: sc-zone-a
          nodeSelector:
            topology.kubernetes.io/zone: zone-a
        - name: zone-b
          storageClass: sc-zone-b
          nodeSelector:
            topology.kubernetes.io/zone: zone-b
        - name: zone-b
          storageClass: sc-zone-b
          nodeSelector:
            topology.kubernetes.io/zone: zone-c

Other proposition

I also have another proposition: automatically create Storage Class when zoneAwareReplication is enabled and add the created Storage Class to the volumeClaimTemplate spec automatically.

What do you think ?

Checklist

• [ ] Tests updated
• [ ] Documentation added
• [ ] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[CLAassistant]

All committers have signed the CLA.

[pstibrany]

The CHANGELOG has just been cut to prepare for the next Mimir release. Please rebase main and eventually move the CHANGELOG entry added / updated in this PR to the top of the CHANGELOG document. Thanks!

[blazarecki]

Hi,@pstibrany thanks for the notification, it's rebased now.

[blazarecki]

Hi @dimitarvdimitrov I apply your changes proposals. Thanks for the feedbacks.