Documentation setting up prometheus and grafana dashboard

[shzshi]

Is this a BUG REPORT or FEATURE REQUEST?:

What happened: Looking for documentation on setting up grafana dashboard for kubeless, i have grafana and prometheus installed locally and also added the dashboard json , still I cannot see the data updated. So wanted to debug the issue.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Kubeless version (use kubeless version):
• Cloud provider or physical cluster:

[jsnowacki]

I have a similar issue. I've setup Prometheus via helm using this instructions
https://itnext.io/kubernetes-monitoring-with-prometheus-in-15-minutes-8e54d1de2e13 and the dashboards for kubernetes work fine, but I do not see anything for Kubeless. I deployed the sample dashboard from documentation and there is nothing shown. Also, in Prometheus UI I don't see any targets that would resemble Kubeless functions.

[Leibniz137]

The metrics I currently see in prometheus using kubeless version v1.0.0-alpha.5 and the python runtime are of the form function_*.

Specifically:

[hmalhi]

The metrics I currently see in prometheus using kubeless version v1.0.0-alpha.5 and the python runtime are of the form function_*.

Specifically:

I'm also not able to see the kubeless function metrics values in prometheus. But i can ping the api and get data - kubectl get --raw "/api/v1/namespaces/default/services/hello:http-function-port/proxy/metrics"

HELP function_calls_total Number of calls to user function

TYPE function_calls_total counter

function_calls_total{method="POST"} 44765.0

TYPE function_calls_created gauge

function_calls_created{method="POST"} 1560374160.581387

HELP function_failures_total Number of exceptions in user function

TYPE function_failures_total counter

function_failures_total{method="POST"} 0.0

TYPE function_failures_created gauge

function_failures_created{method="POST"} 1560374160.581406

HELP function_duration_seconds Duration of user function in seconds

TYPE function_duration_seconds histogram

function_duration_seconds_bucket{le="0.005",method="POST"} 41.0 function_duration_seconds_bucket{le="0.01",method="POST"} 19972.0

Seems there is some config missing or incorrect? Any help?

[alanmoment]

I got problem too. But my cluster version is 1.14.6 and kubeless version is 1.0.5

[alanmoment]

I solved this problem below add scrape in prometheus

prometheus:
  prometheusSpec:
    additionalScrapeConfigs:
    - job_name: 'kubeless'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: (.+):(?:\\d+);(\\d+)
        replacement: ${1}:${2}
        target_label: __address__
      - source_labels: [__meta_kubernetes_pod_label_function]
        action: replace
        regex: (.+)
        replacement: ${1}
        target_label: function