[Bug]: lost telemetry after upgrade to v0.50.7

[scirner22]

Is there an existing issue for this?

• [X] I have searched the existing issues

What happened?

After upgrading from v0.46.13 v0.50.7 we seem to have lost some cosmos layer telemetry.

Configuration

# config.toml
[instrumentation]
prometheus = true
prometheus_listen_addr = ":26660"
max_open_connections = 3
namespace = "cometbft"

# app.toml
[telemetry]
service-name = ""
enabled = true
enable-hostname = true
enable-hostname-label = true
enable-service-label = false
prometheus-retention-time = 60
global-labels = [
  [
    "chain_id",
    "pio-mainnet-1"
  ]
]

Metrics before upgrade

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 75520    0 75520    0     0   363k      0 --:--:-- --:--:-- --:--:--  365k
# HELP store_iavl_commit store_iavl_commit
# TYPE store_iavl_commit summary
store_iavl_commit{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.5"} 0.37117999792099
store_iavl_commit{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.9"} 0.6181100010871887
store_iavl_commit{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.99"} 2.5882089138031006
store_iavl_commit_sum{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1"} 258902.27967266366
store_iavl_commit_count{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1"} 677424
# HELP store_iavl_delete store_iavl_delete
# TYPE store_iavl_delete summary
store_iavl_delete{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.5"} 0.02745000086724758
store_iavl_delete{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.9"} 0.029260000213980675
store_iavl_delete{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.99"} 0.029260000213980675
store_iavl_delete_sum{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1"} 72579.197542749
store_iavl_delete_count{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1"} 76053
# HELP store_iavl_get store_iavl_get
# TYPE store_iavl_get summary
store_iavl_get{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.5"} 0.002409999957308173
store_iavl_get{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.9"} 0.0037499999161809683
store_iavl_get{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.99"} 0.012059999629855156
store_iavl_get_sum{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1"} 2.6048887388131223e+06
store_iavl_get_count{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1"} 3.09507892e+08

Metrics after upgrade

curl http://localhost:26660/metrics  | rg iavl
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  103k    0  103k    0     0   533k      0 --:--:-- --:--:-- --:--:--  534k

Cosmos SDK Version

v0.50.7

How to reproduce?

No response