Adding custom labels to metrics

[rajyan]

Hi, thank you for this great library.

I was wondering if there is a way to add custom labels to client metrics. I can add default attributes with otelsql.WithDefaultAttributes, but is it possible to add semconv.DBStatementKey.String or query arguments for labels?

[rajyan]

Some kind of middleware like https://github.com/nhatthm/otelsql/blob/f4a240f54c53f77caf202e1fe282e59d241f5c6e/options.go#L157-L161 for metrics might be reasonable?

[nhatthm]

What is your use case exactly? 🤔

[rajyan]

What is your use case exactly? :thinking:

I was thinking about

• Count, sum up duration by DBStatementKey to look for the performance by query
• Add labels from context, like http.route to see the stats by other contexts

[nhatthm]

Putting query and its args into the metric labels will drastically increase the cardinality of metrics. You should go with one of these alternatives:

1. Use Traces instead of Metrics. Even with a low sampling rate, it still provides insightful data about the performance of the operations. Some platforms, such as Grafana Tempo or Honeycomb, provide convenience to filter the spans and attributes.
2. Measure the operations at one higher level, for example, the place that calls db.QueryContext(). That will provide the metrics with better context, rather than just the query and the args.

[rajyan]

Putting query and its args

To be clear, I was planning to add query without args, which I think the cardinality should be reasonable (at least in my case). For args I was thinking about more filtered usage, for example, adding labels by specific args that has low cardinality.

Thank you for the alternatives. I'm currently using Grafana Tempo for traces and using Prometheus for metrics. Let me be even more specific, I'm trying to create a query dashboard like AWS RDS Performance Insights https://aws.amazon.com/jp/blogs/database/amazon-rds-performance-insights-is-now-generally-available/ but couldn't find a way to do so. This is why I created this issue (I'm sorry if this is the wrong way to go).

[rajyan]

Searching for Grafana Tempo, https://community.grafana.com/t/dashboard-with-arithmetic-operation-between-two-spans-of-a-trace/88794

For this use-case, the best option is using the metrics-generator 1 with span-metrics 2. This will create latency metrics of your spans, to which you can add attributes that are relevant for you, and then calculate the difference in latency.

maybe this can work in my use case?

[rajyan]

@nhatthm

Created a sample query dashboard in Grafana that can be achieved by this feature. Hope this make sense.

Also, I'm interested in implementing this if you think it's a good idea

[nhatthm]

Searching for Grafana Tempo, https://community.grafana.com/t/dashboard-with-arithmetic-operation-between-two-spans-of-a-trace/88794

For this use-case, the best option is using the metrics-generator 1 with span-metrics 2. This will create latency metrics of your spans, to which you can add attributes that are relevant for you, and then calculate the difference in latency.

maybe this can work in my use case?

I think you can try that.

---

Regarding the original question, this library isn't the right tool for building dashboards like AWS Performance Insights. You will need an exporter that gathers data directly from the database servers. Try to search for XXX prometheus exporter, e.g Postgres exporter.

[rajyan]

Thank you for you for your suggestions.

I think you can try that.

I was able to achieve something similar using this method as well, just to let you know. Although, it is not as flexible as setting those attributes in otelsql.

Regarding the original question, this library isn't the right tool for building dashboards like AWS Performance Insights.

Maybe I'm not understanding the concept of OpenTelemetry metrics correctly, but what is the supposed use case of the otelsql metrics then?

The differences that I think collecting data from otelsql instead of prometheus is that

• The metrics table in https://github.com/nhatthm/otelsql/issues/222#issuecomment-1676494723 has links to traces
• The metrics has contexts that can only obtained from application. Same queries might be called in different endpoints, and we can add those information
• The observations won't depend on the database driver. We can switch from MySQL to PostgreSQL easily