exporter/prometheus: add an example of grafana dashoard of usage metrics from ochttp/ocgrpc

[inotnako]

Hi !

Opencensus expose metrics like *_bucket with count for each delay bucket and others metrics. And I tried build grafana dash from datasource prometheus on that metrics and not everything went smoothly.

Please give some example dashboard where I will look rate ops/sec and it will correctly and by pathTagKey too.

Better way if we will starting create one dashboard on exposed metrics for ochttp/ocgrpc

[odeke-em]

Hello there @antonikonovalov! Thank you filing this issue and welcome to OpenCensus. Firstly sorry for the radio silence, we've been quite swamped.

Great question. So would you mind articulating on what you mean by

Opencensus expose metrics like *_bucket with count for each delay bucket and others metrics. And I tried build grafana dash from datasource prometheus on that metrics and not everything went smoothly.

I just saw this issue and hacked up an end-to-end runnable server using ochttp{Client, Server} like this

main.go

package main

import (
	"io/ioutil"
	"log"
	"math/rand"
	"net/http"
	"net/http/httptest"
	"strings"
	"time"

	"go.opencensus.io/exporter/prometheus"
	"go.opencensus.io/plugin/ochttp"
	"go.opencensus.io/stats/view"
	"go.opencensus.io/trace"
)

func main() {
	trace.ApplyConfig(trace.Config{DefaultSampler: trace.AlwaysSample()})

	if err := view.Register(ochttp.DefaultServerViews...); err != nil {
		log.Fatalf("Failed to register ochttp Server views: %v", err)
	}
	if err := view.Register(ochttp.DefaultClientViews...); err != nil {
		log.Fatalf("Failed to register ochttp Client views: %v", err)
	}
        view.SetReportingPeriod(300 * time.Millisecond)

	cst := httptest.NewServer(&ochttp.Handler{
		Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			_, _ = ioutil.ReadAll(r.Body)
			_ = r.Body.Close()
			w.Write([]byte("Pong"))
		}),
	})
	defer cst.Close()

	pe, err := prometheus.NewExporter(prometheus.Options{
		Namespace: "tutorial",
	})
	if err != nil {
		log.Fatalf("Failed to create Prometheus exporter: %v", err)
	}
        view.RegisterExporter(pe)

	go func() {
		mux := http.NewServeMux()
		mux.Handle("/metrics", pe)
		if err := http.ListenAndServe(":8888", mux); err != nil {
			log.Fatalf("Failed to serve Prometheus handler: %v", err)
		}
	}()

	client := &http.Client{Transport: new(ochttp.Transport)}
	for {
		req, _ := http.NewRequest("POST", cst.URL, strings.NewReader("Ping"))
		res, err := client.Do(req)
		if err != nil {
			log.Fatalf("Failed to make http request: %v", err)
		}
		_, _ = ioutil.ReadAll(res.Body)
		_ = res.Body.Close()

		ir := rand.Intn(1000)
		<-time.After(time.Millisecond * time.Duration(ir))
	}
}

prometheus.yaml

global:
  scrape_interval: 5s

  external_labels:
    monitor: 'demo' 

scrape_configs:
  - job_name: 'demo'

    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:8888']

Running it

In one terminal, run the main.go code

$ go run main.go 

In another terminal start Prometheus

$ prometheus --config.file prometheus.yaml

Start Grafana

brew services start grafana

Walk through

Hope this helps, but really Grafana dashboards are tertiary tools that work with Prometheus, I am not sure if we can include such samples in the Go repository, perhaps the website?

[inotnako]

Hello @odeke-em, thank you for the answer and example.

Hope this helps, but really Grafana dashboards are tertiary tools that work with Prometheus, I am not sure if we can include such samples in the Go repository, perhaps the website?

agree with you - dashboards are tertiary tools maybe we can collect them here - https://github.com/census-ecosystem

Great question. So would you mind articulating on what you mean by

Excellent! You have all the illustrative examples in the screenshots. Sorry for simply question, but what can you understand on the current diagram in the panel?

What benefit can be derived from this metrics? for me the use of these metrics does not look transparent. Maybe I missed something...

of course, I can find more information for customization and use metrics on an internet.

And maybe we should know why needed a buckets aggregation on that metric and which cases it can be helpful for show critical moments or some tends.

Sorry, maybe that issue sounds like quibbling.

I just want metrics to be more usable out of the box and I should know how and for what used them.

On that illustration, I can understand what happened.

[semistrict]

There is ongoing work to do this for Graphite here: https://github.com/census-ecosystem/opencensus-go-exporter-graphite/pull/12

I agree that we need it for Prometheus too - I hope it will be similar.

[emcfarlane]

Was looking for examples too and thought this might help:

histogram_quantile(0.9, sum(rate(grpc_io_server_server_latency_bucket{kubernetes_name=~"$app"}[2m])) by (grpc_server_method, le))