Unable to specify settings for prometheus in ConfigMap

[VincentVerweij]

I am currently configuring the ConfigMap that is used by the OMS-agent pods. What I want to achieve is sending Prometheus metrics to a log analytics workspace. For this I am following this Microsoft docs page. On that page we can see this:

prometheus.io/scrape: "true"
prometheus.io/path: "/mymetrics"
prometheus.io/port: "8000"
prometheus.io/scheme: "http"

And then in the table under Cluster-wide we have keys like these:

• prometheus.io/scrape
• prometheus.io/path
• ...

So, my understanding is that a user can specify what port e.g. the OMS-agent has to look at in the annotations of an application pod. In my case, I have a pod that has an annotation: prometheus.io/port=8900. And the default that is mentioned in the documentation is 9102.

I tried to specify the following TOML in the ConfigMap:

prometheus-data-collection-settings: |-
    [prometheus_data_collection_settings.cluster]
        interval = "1m"
        fieldpass = ["platform_user_sessions", "platform_connection_bus"]
        monitor_kubernetes_pods = true
        monitor_kubernetes_pods_namespaces = ["dev-group-apps"]
        prometheus.io/port = 8900

Once the ConfigMap is read by the OMS-agent, I get the following error in the logs of the pod:

"config::error::Exception while parsing config map for prometheus config: \nparse error on value \"/\" (error), using defaults, please check config map for errors"

When commenting the prometheus.io/port = 8900, it is parsed successfully.

I started to look in the source code to find the error, and what it does when it successfully parses the configmap. There I bumped into these statements: https://github.com/microsoft/Docker-Provider/blob/0903e864665d1d606921fc10c5ab75a6ae66b83e/build/common/installer/scripts/tomlparser-prom-customconfig.rb#L144-L154

There is no prometheus read from the parsedConfig so I am definitely doing something wrong.

How are we able to specify that the OMS-agent has to look for a different port in the prometheus.io/port annotation, or is my understanding of this completely wrong?