:boom: Replace OTel Prometheus Exporter
This PR makes some significant alterations to the metrics abstractions and replaces the OTel Prometheus exporter with a custom one, since theirs is no longer maintained. The custom one is essentially a copy-paste of Prom's own first-party lib, except with some modifications made to allow for the registration of metrics with overlapping labels (this is fully permitted by Prom for ingestion, mind you, just prevented client side. They have their reasons for that, which are legitimate, but taking that option away would require fully breaking our API).
Breaking
- Metric
Arc<dyn XXX>are now concrete types -
with_attributesmethods now return Results, just no good way around this. Dealing with it in lang bridges should be fairly easy though since the possibility of throwing something generally existed already.
New
- All the metrics now have
addsorrecords(as opposed to justadd) which can be used (afterwith_attributes, typically) to record without passing labels again. This is more efficient for the prom backend. They needed a different name to avoid turbofish disambiguation nonsense.
I did some testing to ensure performance isn't materially different here, and it's not in real-world situations (ex: the workflow_load integration test has no difference in overall runtime). That said, the specific bench I made does show that if you record a bajillion prom metrics as fast as you can from different threads, lock contention slows things down by as much as 50%. At the same time, OTel's implementation of what actually happens when you scrape prom metrics is to just basically create every single metric on demand from scratch and fill in the data, which is definitely slower than what happens now - so actual scraping should be quite a bit cheaper. TLDR: I don't have any reason to believe this would have any real impact, for anyone unless they're using custom metrics with the prom exporter and are just absolutely hammering away.
If further perf improvements are desired in the future, we could add a bind_with_schema kind of API which would bind a metric to a certain set of labels, and allow recording on that metric with different label values without going through a lock.
Closes https://github.com/temporalio/sdk-core/issues/908 Closes https://github.com/temporalio/sdk-core/issues/882