Configuring Prometheus metric reporter

[aleksandr-spb]

Good day. We use Flink Prometheus metric reporter (https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/metrics.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter) in conjunction with Prometheus Operator ServiceMonitor (https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md#related-resources) to collect Flink cluster metrics. But when we tried to swithc to flinkk8soperator we encountered some difficulties.

To collect metrics we need to expose metric reporter port 9249 on both services- JobManager and TaskManager, but FlinkApplication configuration does not allow configuring container ports, and flinkk8soperator creates a Service only for JobManager.

Can we configure FlinkApplication to collect metrics without changing the source code of flinkk8soperator?

[anandswaminathan]

@aleksandr-spb Can you share more information on the difficulties?

Is metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter not working? Correct me if my understanding is incorrect. Based on doc- https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.14/: the ports is primarily informational. Not specifying a port here does not prevent that port from being exposed

[aleksandr-spb]

1. PrometheusReporter is working. But we cant access it through Prometheus Operator ServiceMonitor. ServiceMonitor needs metric endpoints that are provided by the Services. We need to explicitly configure port 9249 on the corresponding Service (JobmanagerService and TaskManagerService).
2. FlinkK8SOperator only creates a Service for Jobmanager, but we need to get metrics from TaskManager also through the Service. I create branch in fork - https://github.com/aleksandr-spb/flinkk8soperator/tree/supportingServiceMonitor. It works, but I'm not sure if my decision is correct

[anandswaminathan]

@aleksandr-spb @srleyva

For (1) I think we can think of adding "additionalServicePorts" either in CRD, or operator config that can be propagated to the service.

For (2) Just to understand more - essentially we need to create a separate TaskManager service - this service need not be versioned like Jobmanager service - and can live for the lifetime of the application (believe this is what you have done in your repo). Also we can throw this behind a config flag.

Is that correct ? If that is the case, we can achieve all this without touching the CRD right ?

[aleksandr-spb]

That is correct.

[iravid]

Posted this on the Slack channel, but for anyone interested, here's a PodMonitor spec that can monitor applications deployed by the operator. This can serve as a workaround until changes are made in the services that the operator deploys:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: flink-podmonitor
  namespace: <application namespace>
spec:
  namespaceSelector:
    matchNames:
    - <application namespace>
  selector:
    matchLabels:
      <application labels>
  podMetricsEndpoints:
    - path: /
      interval: 30s
      relabelings:
      # The Kubernetes service discovery will create a target for every port on the
      # Flink containers. So first, keep only the targets for the RPC port.
      - sourceLabels:
        - __meta_kubernetes_pod_container_port_number
        action: keep
        regex: "6123"
      # Next, we don't actually want the RPC port. So replace the port number for
      # the container port label and the endpoint label to 9249.
      - targetLabel: __meta_kubernetes_pod_container_port_number
        replacement: "9249"
      - targetLabel: endpoint
        replacement: "9249"
      # Finally, override the special __address__ label that specifies what IP:Port
      # Prometheus should scrape with the right value.
      - sourceLabels:
        - __meta_kubernetes_pod_ip
        - __meta_kubernetes_pod_container_port_number
        separator: ":"
        targetLabel: __address__

[srleyva]

@iravid I was looking at using this, however, I do think the pods needs to expose the port as well for it to be scraped through the containers[*].ports option. The workaround I was looking at was creating a service outside of the operator. This did not work because the pod does not expose the port as well.

[iravid]

@srleyva The port does not need to be specified in the ContainerSpec for Prometheus to be able to scrape it. It just needs to be there to be discovered by the kubernetes_sd. This is why the trickery with the relabelings is needed.

[srleyva]

It worked. I had the metric port misconfigured. Thanks @iravid! This work around is sufficient for us.

[rehevkor5]

we need to create a separate TaskManager service

You do not need a Service, if that's what you're saying. You just need to allow the Prometheus metrics scraping port to be exposed on the TaskManager containers. Prometheus periodically sends a metrics request to each individual pod.

[HunterEl]

@rehevkor5 apologies for commenting on an older issue, but since it's not closed I feel like it's still open season. I'm hitting this issue now with a deployed of Flink Operator on Kube. Our Prometheus uses service-discovery over hard-coding, and specifying the pod with prometheus annotations and the prometheus port flink config is not enough. What you described is what we need. The prometheus service discovery requires that the port is also open on the container. I've found trying to open up this port on the JobManager and TaskManager containers, and serve back metrics, to be non-trivial and would love some guidance if you have any (or from anyone else reading this). Thanks!

[srleyva]

@HunterEl Have you tried the re-labeling solution above? It's been a minute since I've messed with this but I seem to remember it working. Though not entirely sure how as its been a minute.

[HunterEl]

@srleyva Let me give it a go! I didn't think it would work for our problem, but It's worth a shot!

[HunterEl]

@srleyva Ah, I tried this out and didn't realize that PodMonitor was apart of the core-os prom operator. We are running our own prometheus setup, so I will need to find an analog for pod/container relabeling. Will most likely be using mutating admission webhook or something similar.

[srleyva]

In that case you may need this feature added to the operator. Last I check the CRD doesn’t expose the option to open ports on flink pods. The Prometheus instances needs to be able to scrape on a configured port. @HunterEl

[rogaha]

+1

[jaroslaw-osmanski]

@aleksandr-spb, your branch with service for TM looks good to be merged to mainline. Is there some reason why you cannot push that here?

[rogaha]

@aleksandr-spb have you submitted the PR?

[rogaha]

@anandswaminathan are you guys using Prometheus for metrics collection?

[srivatsav]

Hey @iravid your solution of PodMonitor is working great when I am port forwarding on the pod, I am able to see the metrics as well. However, the kube-prometheus operator which we are using is still not discovering the podmonitor and I am still unable to see the flink metrics in my prometheus. Any clue ? The podMonitorNamespaceSelector: {} is configured this way in prometheus operator. Does it mean it will look for in same namespace only ? My PodMonitor and operator are in diff namespaces.

[lebenitza]

@srivatsav this might help you:

• https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml#L2319
• https://github.com/prometheus-operator/kube-prometheus/issues/181

[srivatsav]

@lebenitza appreciate it. However, I came up with a small work around of writing another service component as below and exposing a metric port from the container.

in my Dockerfile - echo "metrics.reporter.prom.port: 9025" >> "$FLINK_HOME/conf/flink-conf.yaml";

in my Service.yml -

apiVersion: v1
kind: Service
metadata:
  name: flinkapp-service
  labels:
    svc: flinkapp-service
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 9025
      protocol: TCP
      name: custom-metric-port
  selector:
    app: work-manager

Now adding a service monitor on top of this service got me all of my metrics :-)