Export exponent in prometheus metrics

[DavidNix]

Problem

To detect low balances, we have to join other metrics with cosmos_relayer_wallet_balance to get the exponent. If we forget to add the joined metrics, we may miss low balances.

cosmos_relayer_wallet_balance already has a denom label.

~~Could we get the exponent as well? E.g. cosmos_relayer_wallet_balance{denom="uatom", exponent="6"}.~~

Edit: Apologies, the above is not how prometheus works. We need a new metric so we can do arithmetic on the metric values. You can't pivot a label into a value AFAIK. Labels are purely for categorization.

So a possible solution is a new metric. We must have the denom to enable the join.

E.g.

cosmos_relayer_denom_exponent{denom="uatom"} 6

If the relayer always exports the exponent, we would be able to fire low balance alerts accurately without needing to join on other metrics we have to create outside of the relayer.

Limitations

I don't know if it's possible or practictal to query for the exponent within the relayer. If not or if you have to add exponent as part of the config, I would suggest closing this story.

Given the need for a 2nd metric, I'm questioning the value of this story.