grafana dashboard example

[benitogf]

Would someone be interested in this addition to the wiki/docs? or at least sharing some nice graph query/config, have been working on it but there's not much information available or at least not related to tusd specifically.

[Acconut]

I am not aware of someone who uses Grafana to monitor tusd but this does not mean that it has not been done :) We would be very pleased if you would like to contribute this information to tusd's documentation, in particular since we are no Grafana experts on our own. Feel free to ask for our assistance when needed!

[benitogf]

Thanks for the enthusiasm :) my goal is to create a chart to monitor request duration for now, found that tusd doesn't have the instrumentation needed for this kind of chart so I've been investigating on how to add them, few useful resources found so far:

duration collector (in node) doc for the instrumentation of duration in go a very nice post on tracking request duration

So next step is to try to implement it on tusd, would a PR to add this be welcomed?

[Acconut]

Thanks for the enthusiasm :)

We are always pleased to help people contribute to the tus ecosystem :)

my goal is to create a chart to monitor request duration for now

What is your definition for request duration? Do you want to include all requests into this metric (GET, HEAD, POST, PATCH)? GET and HEAD request should be relatively fast but PATCH requests may range from less than a second to a few hours, so I am not sure how you would use this information. Could you elaborate more in this idea?

[benitogf]

Very good question indeed the duration of each request will not provide meaningful information about the process, nevertheless the same duration instrumentation could be applied to:

upload duration

between start and complete (or fail)

download duration

GET requests could be instrumented directly

[benitogf]

  histogram := prometheus.NewHistogramVec(prometheus.HistogramOpts{
    Name: "process_duration_seconds",
    Help: "Time taken to process a file",
  }, []string{"action"})

  start := time.Now()

  // Process here
  // how do we keep the start time related to the file being processed?
  // could be an in-memory hash with uid -> startTime?

  duration := time.Since(start)
  action := "upload" // upload or download
  histogram.WithLabelValues(action).Observe(duration.Seconds())

[Acconut]

how do we keep the start time related to the file being processed? could be an in-memory hash with uid -> startTime?

An in-memory storage is not suitable for a setup of multiple tusd instances, where requests for the same upload may be handled by different servers. Therefore you should not store this information in memory. The solution would be to add the upload start time to its metadata which can be accessed by all instances.

However, I am still very interested in this, but I haven't yet fully figured out how to properly use histograms and summaries with Prometheus. So I need some more time there :)

[benitogf]

start time in the metadata is a much better idea :)

[Acconut]

I am going to work on this next week and keep you updated.

[geor-g]

Could people who make use of Grafana share their dashboards?

[Acconut]

In the past five years we actually have started using Grafana to monitor tusd. But I am not sure how we could share some details about our dashboard? Maybe @kvz knows a good way?

[geor-g]

Could you export it, and make the Json available?

[kvz]

We have a bunch of other metrics on our dashboard as well, but perhaps we can create a tusd specific dashboard (initially in addition, not as a replacement I would say), export that, evolve it on GitHub with the community and port back patches up- & down stream?

[kvz]

I think there also is some Grafana hosted library where we could have it indexed I believe

[geor-g]

We have a bunch of other metrics on our dashboard as well, but perhaps we can create a tusd specific dashboard (initially in addition, not as a replacement I would say), export that, evolve it on GitHub with the community and port back patches up- & down stream?

Sounds good. :+1:

[geor-g]

I think there also is some Grafana hosted library where we could have it indexed I believe

This https://grafana.com/grafana/dashboards/ probably?

[kvz]

Yes apologies, I was on mobile earlier, but that is exactly what I mean :) After sharing with the community and processing feedback, we should put it up there too i think.

[Acconut]

This https://grafana.com/grafana/dashboards/ probably?

That's interesting! Is there any documentation on how we can publish there?

[geor-g]

That's interesting! Is there any documentation on how we can publish there?

@Acconut Didn't get a notfication back then, hence the delay: on the linked page there is the following:

Share your dashboards

Export any dashboard from Grafana 3.1 or greater and share your creations with the community.

[Acconut]

Thank you, we will try to look into this in the future