[grafana] Sidecar provider breaks if folder annotation value is changed

[FelipeEmerim]

Hi,

We have an issue using the sidecar dashboard provider with the following configuration:

sidecar:
  # Log level. Can be one of: DEBUG, INFO, WARN, ERROR, CRITICAL.
  logLevel: ERROR
  dashboards:
    env: {}
    enabled: true
    SCProvider: true
    # label that the configmaps with dashboards are marked with
    label: grafana-dashboard
    # value of label that the configmaps with dashboards are set to
    labelValue: "1"
    # folder in the pod that should hold the collected dashboards (unless `defaultFolderName` is set)
    folder: /tmp/dashboards
    # The default folder name, it will create a subfolder under the `folder` and put dashboards in there instead
    defaultFolderName: null
    # Namespaces list. If specified, the sidecar will search for config-maps/secrets inside these namespaces.
    # Otherwise the namespace in which the sidecar is running will be used.
    # It's also possible to specify ALL to search in all namespaces.
    searchNamespace:
      - monitoring
    # Method to use to detect ConfigMap changes. With WATCH the sidecar will do a WATCH requests, with SLEEP it will list all ConfigMaps, then sleep for 60 seconds.
    watchMethod: SLEEP
    # search in configmap, secret or both
    resource: configmap
    # If specified, the sidecar will look for annotation with this name to create folder and put graph here.
    # You can use this parameter together with `provider.foldersFromFilesStructure`to annotate configmaps and create folder structure.
    folderAnnotation: grafana-dashboard-folder
    # Absolute path to shell script to execute after a configmap got reloaded
    script: null
    provider:
      # name of the provider, should be unique
      name: sidecarProvider
      # orgid as configured in grafana
      orgid: 1
      # folder in which the dashboards should be imported in grafana
      folder: ''
      # type of the provider
      type: file
      # disableDelete to activate a import-only behaviour
      disableDelete: false
      # allow updating provisioned dashboards from the UI
      allowUiUpdates: false
      # allow Grafana to replicate dashboard structure from filesystem
      foldersFromFilesStructure: true
    # Additional dashboard sidecar volume mounts
    extraMounts: []
    # Sets the size limit of the dashboard sidecar emptyDir volume
    sizeLimit: {}

This works fine most of the time, however if we change the value of the folder annotation on one of the configmaps, the sidecar mounts the dashboard in the new folder but does not remove the old mount. This results in duplicate dashboard errors until we restart the pod.

How to reproduce:

1. Deploy the helm chart configuring the dashboard sidecar provider with a config similar to the above. It is important to use folderAnnotation + foldersFromFileStructure.
2. Create a configmap containing a dashboard and fill folderAnnotation with the value /tmp/dashboards/tomato
3. Open a shell into the sidecar container and access the /tmp/dashboards folder to check the results.
4. Change the existing config map folderAnnotation with the value /tmp/dashboards/potato
5. Open a shell into the sidecar container and access the /tmp/dashboards folder to check the results.
6. See that both potato and tomato are mounted and contain the same dashboard which results in the following logs
7. Optionally, restart the pod to see that in the new pod only the potato folder is mounted.

Is there something wrong with our configuration or is it a problem in the helm chart?

Environment:

• k8s: 1.23.8
• grafana: 9.0.4
• helm chart: grafana
• helm chart version: 6.32.7