How can I run this in kubernetes?

[jaya-rama]

[denisok]

we have a feature waiting to be implemented #265. Theoretically you could prepare container by yourself and run it as a sidecar container.

[denisok]

there is a container published now with v0.20.9 https://github.com/percona/mongodb_exporter/releases/tag/v0.20.9

You can use that to run on k8s. You just need to run it as a sidecar along with monitored service.

[jonaskint]

Another approach for this could be something similar on how Istio does it's Envoy sidecar injection. They work with a configmap that specifies the actual sidecar template. The percona mongodb operator could then add those based on the labels that are attached to the statefulsets that run a mongo cluster.

[kaykhan]

Can anyone help me with creating a confguration on how to add this as a sidecar to the prometheus mongodb operator. There is some documentation on how to define a sidecar - https://docs.percona.com/percona-operator-for-mongodb/sidecar.html

Im fairly new to this, so other than setting the image: percona/mongodb_exporter:0.36 is there anything else i need to do?

spec:
  replsets:
    sidecars:
    - image: percona/mongodb_exporter:0.36
      command: ["/bin/sh"] // assuming this needs to be configured?
      args: ["-c", "while true; do echo echo $(date -u) 'test' >> /dev/null; sleep 5; done"] // assuming this needs to be configured?
      name: rs-sidecar-0

[denisok]

@kaykhan if you want to monitor it with Percona operator for MongoDB - please consider PMM. Operator already have a sidecar for that.

In you example, you need to change the command, as well as args. You also need to specify port to open so extrenal prometheus could query exporter.

spec:
  replsets:
    sidecars:
    - image: percona/mongodb_exporter:0.37
      command: ["/mongodb_exporter"] 
      args: ["--mongodb.uri=mongodb://127.0.0.1:17001"] 
      ports:
      - exporter: 9216
      name: rs-sidecar-0

something like that but I didn't verify it.

[kaykhan]

@kaykhan if you want to monitor it with Percona operator for MongoDB - please consider PMM. Operator already have a sidecar for that.

In you example, you need to change the command, as well as args. You also need to specify port to open so extrenal prometheus could query exporter.

spec:
  replsets:
    sidecars:
    - image: percona/mongodb_exporter:0.37
      command: ["/mongodb_exporter"] 
      args: ["--mongodb.uri=mongodb://127.0.0.1:17001"] 
      ports:
      - exporter: 9216
      name: rs-sidecar-0

something like that but I didn't verify it.

ye i got it working, i'll leave the solution here incase anycome comes across this https://forums.percona.com/t/how-to-collect-metrics-and-ship-to-existing-prometheus-grafana-setup/21616

What grafana dashboards work with mongodb_exporter?

[denisok]

https://github.com/percona/grafana-dashboards/tree/main/dashboards/MongoDB has set of Grafana dashboards.

You probably need to run https://github.com/percona/grafana-dashboards/blob/main/misc/convert-dash-from-PMM.py to get it worked for the vanila prom (standard labels).

We are trying to PoC some solution to push it as a mixin in this repo, but that is not done yet.

[kaykhan]

https://github.com/percona/grafana-dashboards/tree/main/dashboards/MongoDB has set of Grafana dashboards.

You probably need to run https://github.com/percona/grafana-dashboards/blob/main/misc/convert-dash-from-PMM.py to get it worked for the vanila prom (standard labels).

We are trying to PoC some solution to push it as a mixin in this repo, but that is not done yet.

Where do i run that script?

I managed to install the dashboards and majority of it is working.

[denisok]

locally. Something like convert-dash-from-PMM.py MongoDB_Cluster_Summary.json

[kaykhan]

locally. Something like convert-dash-from-PMM.py MongoDB_Cluster_Summary.json

I get some errors, not sure why. Would be cool if someone could convert them and upload the .json as CONVERTED_MongoDB_Cluster_Summary.json or something.

./convert-dash-from-PMM.py MongoDB_Cluster_Summary.json
import-im6.q16: attempt to perform an operation not allowed by the security policy `PS' @ error/constitute.c/IsCoderAuthorized/421.
import-im6.q16: attempt to perform an operation not allowed by the security policy `PS' @ error/constitute.c/IsCoderAuthorized/421.
import-im6.q16: attempt to perform an operation not allowed by the security policy `PS' @ error/constitute.c/IsCoderAuthorized/421.
./convert-dash-from-PMM.py: 24: services: not found
./convert-dash-from-PMM.py: 27: Syntax error: "(" unexpected

[anton-bystrov]

@kaykhan Hello, try Python 2 please.

[kaykhan]

@kaykhan Hello, try Python 2 please.

Yes it ran successfully i think using python 2

python convert-dash-from-PMM.py MongoDB_Cluster_Summary.json
Dashboard: MongoDB_Cluster_Summary.json
MongoDB service dashboard is detected
 <<<< label_values(mongodb_up{cluster="$cluster"},service_name)
 >>>> label_values(mongodb_up{cluster="$cluster"},instance)

 <<<< service_name
 >>>> instance

 <<<< label_values(pg_stat_database_tup_fetched{service_name=~"$service_name",datname!~"template.*|postgres"},datname)
 >>>> label_values(pg_stat_database_tup_fetched{instance=~"$instance",datname!~"template.*|postgres"},datname)

 <<<< label_values(mysql_info_schema_user_statistics_connected_time_seconds_total{service_name="$service_name"},user)
 >>>> label_values(mysql_info_schema_user_statistics_connected_time_seconds_total{instance="$instance"},user)

 <<<< sum by (service_name) (mongodb_connections{cluster="$cluster", state="current"} * on (service_name) group_right avg by (service_name,set) (mongodb_mongod_replset_my_state{cluster="$cluster",set!~"$shard"}/ mongodb_mongod_replset_my_state{cluster="$cluster",set!~"$shard"}))
 >>>> sum by (instance) (mongodb_connections{cluster="$cluster", state="current"} * on (instance) group_right avg by (instance,set) (mongodb_mongod_replset_my_state{cluster="$cluster",set!~"$shard"}/ mongodb_mongod_replset_my_state{cluster="$cluster",set!~"$shard"}))

 <<<< (sum by (service_name) (rate(mongodb_op_counters_total{service_name=~"$service_name",type!="command"}[$interval]) or irate(mongodb_op_counters_total{service_name=~"$service_name",type!="command"}[5m])) * on (service_name) group_right avg by (service_name,set) (mongodb_mongod_replset_my_state{cluster="$cluster",set!~"$shard"}/ mongodb_mongod_replset_my_state{cluster="$cluster",set!~"$shard"}))
 >>>> (sum by (instance) (rate(mongodb_op_counters_total{instance=~"$instance",type!="command"}[$interval]) or irate(mongodb_op_counters_total{instance=~"$instance",type!="command"}[5m])) * on (instance) group_right avg by (instance,set) (mongodb_mongod_replset_my_state{cluster="$cluster",set!~"$shard"}/ mongodb_mongod_replset_my_state{cluster="$cluster",set!~"$shard"}))

 <<<< (sum by (service_name) (rate(mongodb_op_counters_total{service_name=~"$service_name",type!="command"}[$interval]) or irate(mongodb_op_counters_total{service_name=~"$service_name",type!="command"}[5m])) * on (service_name) group_right avg by (service_name) (avg by (service_name) (mongodb_mongos_db_collections_total{cluster="$cluster"}) / avg by (service_name) (mongodb_mongos_db_collections_total{cluster="$cluster"})))
 >>>> (sum by (instance) (rate(mongodb_op_counters_total{instance=~"$instance",type!="command"}[$interval]) or irate(mongodb_op_counters_total{instance=~"$instance",type!="command"}[5m])) * on (instance) group_right avg by (instance) (avg by (instance) (mongodb_mongos_db_collections_total{cluster="$cluster"}) / avg by (instance) (mongodb_mongos_db_collections_total{cluster="$cluster"})))

 <<<< (sum by (service_name) (rate(mongodb_op_counters_total{service_name=~"$service_name",type!="command"}[$interval]) or irate(mongodb_op_counters_total{service_name=~"$service_name",type!="command"}[5m])) * on (service_name) group_right avg by (service_name,set) (mongodb_mongod_replset_my_state{cluster="$cluster",set=~"$shard"}/ mongodb_mongod_replset_my_state{cluster="$cluster",set=~"$shard"}))
 >>>> (sum by (instance) (rate(mongodb_op_counters_total{instance=~"$instance",type!="command"}[$interval]) or irate(mongodb_op_counters_total{instance=~"$instance",type!="command"}[5m])) * on (instance) group_right avg by (instance,set) (mongodb_mongod_replset_my_state{cluster="$cluster",set=~"$shard"}/ mongodb_mongod_replset_my_state{cluster="$cluster",set=~"$shard"}))

 <<<< sum by (set) (avg by (service_name,set) (mongodb_connections{cluster="$cluster", state="current"}) * on (service_name) group_right avg by (service_name,set) (mongodb_mongod_replset_my_state{cluster="$cluster"}/ mongodb_mongod_replset_my_state{cluster="$cluster"}))
 >>>> sum by (set) (avg by (instance,set) (mongodb_connections{cluster="$cluster", state="current"}) * on (instance) group_right avg by (instance,set) (mongodb_mongod_replset_my_state{cluster="$cluster"}/ mongodb_mongod_replset_my_state{cluster="$cluster"}))

 <<<< sum by () (avg by (service_name,set) (mongodb_connections{cluster="$cluster", state="current"}) * on (service_name) group_right avg by (service_name,set) (mongodb_mongod_replset_my_state{cluster="$cluster"}/ mongodb_mongod_replset_my_state{cluster="$cluster"}))
 >>>> sum by () (avg by (instance,set) (mongodb_connections{cluster="$cluster", state="current"}) * on (instance) group_right avg by (instance,set) (mongodb_mongod_replset_my_state{cluster="$cluster"}/ mongodb_mongod_replset_my_state{cluster="$cluster"}))

 <<<< sum by (shard) (avg by (service_name,shard) (mongodb_connections{cluster="$cluster", state="current"}) * on (service_name) group_right avg by (service_name,shard) (mongodb_mongos_db_collections_total{cluster="$cluster"} / mongodb_mongos_db_collections_total{cluster="$cluster"}))
 >>>> sum by (shard) (avg by (instance,shard) (mongodb_connections{cluster="$cluster", state="current"}) * on (instance) group_right avg by (instance,shard) (mongodb_mongos_db_collections_total{cluster="$cluster"} / mongodb_mongos_db_collections_total{cluster="$cluster"}))

 <<<< sum by (service_name) (max_over_time(mongodb_connections{cluster="$cluster",service_name=~"$service_name",state="current"}[$interval]) or
max_over_time(mongodb_connections{cluster="$cluster",service_name=~"$service_name",state="current"}[5m]))
 >>>> sum by (instance) (max_over_time(mongodb_connections{cluster="$cluster",instance=~"$instance",state="current"}[$interval]) or
max_over_time(mongodb_connections{cluster="$cluster",instance=~"$instance",state="current"}[5m]))

 <<<< sum by (service_name) (max_over_time(mongodb_connections{cluster="$cluster",service_name=~"$service_name",state="available"}[$interval]) or
max_over_time(mongodb_connections{cluster="$cluster",service_name=~"$service_name",state="available"}[5m]))
 >>>> sum by (instance) (max_over_time(mongodb_connections{cluster="$cluster",instance=~"$instance",state="available"}[$interval]) or
max_over_time(mongodb_connections{cluster="$cluster",instance=~"$instance",state="available"}[5m]))

 <<<< sum(sum(mongodb_mongod_metrics_cursor_open{cluster="$cluster", state="total"} or mongodb_mongod_cursors{cluster="$cluster", state="total_open"}) by (service_name) * on (service_name) group_right mongodb_mongod_replset_my_state{cluster="$cluster"} / mongodb_mongod_replset_my_state{cluster="$cluster"}) by (set)
 >>>> sum(sum(mongodb_mongod_metrics_cursor_open{cluster="$cluster", state="total"} or mongodb_mongod_cursors{cluster="$cluster", state="total_open"}) by (instance) * on (instance) group_right mongodb_mongod_replset_my_state{cluster="$cluster"} / mongodb_mongod_replset_my_state{cluster="$cluster"}) by (set)

 <<<< sum(sum(mongodb_mongod_metrics_cursor_open{cluster="$cluster", state="total"} or mongodb_mongod_cursors{cluster="$cluster", state="total_open"}) by (service_name) * on (service_name) group_right mongodb_mongod_replset_my_state{cluster="$cluster"} / mongodb_mongod_replset_my_state{cluster="$cluster"})
 >>>> sum(sum(mongodb_mongod_metrics_cursor_open{cluster="$cluster", state="total"} or mongodb_mongod_cursors{cluster="$cluster", state="total_open"}) by (instance) * on (instance) group_right mongodb_mongod_replset_my_state{cluster="$cluster"} / mongodb_mongod_replset_my_state{cluster="$cluster"})

 <<<< sum by (service_name) (
max_over_time(mongodb_mongod_metrics_cursor_open{cluster="$cluster", state="total"}[$interval]) or
max_over_time(mongodb_mongod_metrics_cursor_open{cluster="$cluster", state="total"}[5m]) or max_over_time(mongodb_mongos_metrics_cursor_open{cluster="$cluster", state="total"}[$interval]) or
max_over_time(mongodb_mongos_metrics_cursor_open{cluster="$cluster", state="total"}[5m]) or
max_over_time(mongodb_mongod_cursors{cluster="$cluster", state="total_open"}[$interval]) or
max_over_time(mongodb_mongod_cursors{cluster="$cluster", state="total_open"}[5m]) or
max_over_time(mongodb_mongos_cursors{cluster="$cluster", state="total_open"}[$interval]) or
max_over_time(mongodb_mongos_cursors{cluster="$cluster", state="total_open"}[5m])
)
 >>>> sum by (instance) (
max_over_time(mongodb_mongod_metrics_cursor_open{cluster="$cluster", state="total"}[$interval]) or
max_over_time(mongodb_mongod_metrics_cursor_open{cluster="$cluster", state="total"}[5m]) or max_over_time(mongodb_mongos_metrics_cursor_open{cluster="$cluster", state="total"}[$interval]) or
max_over_time(mongodb_mongos_metrics_cursor_open{cluster="$cluster", state="total"}[5m]) or
max_over_time(mongodb_mongod_cursors{cluster="$cluster", state="total_open"}[$interval]) or
max_over_time(mongodb_mongod_cursors{cluster="$cluster", state="total_open"}[5m]) or
max_over_time(mongodb_mongos_cursors{cluster="$cluster", state="total_open"}[$interval]) or
max_over_time(mongodb_mongos_cursors{cluster="$cluster", state="total_open"}[5m])
)

 <<<< sum by (type) (rate(mongodb_op_counters_total{cluster="$cluster", type!="command",service_name=~"$service_name"}[$interval]) or
irate(mongodb_op_counters_total{cluster="$cluster", type!="command",service_name=~"$service_name"}[5m]))
 >>>> sum by (type) (rate(mongodb_op_counters_total{cluster="$cluster", type!="command",instance=~"$instance"}[$interval]) or
irate(mongodb_op_counters_total{cluster="$cluster", type!="command",instance=~"$instance"}[5m]))

 <<<< max by (set) (max(max_over_time(mongodb_mongod_replset_member_replication_lag{cluster="$cluster"}[$interval]) > 0) by (service_name,set) or max(max_over_time(mongodb_mongod_replset_member_replication_lag{cluster="$cluster"}[5m]) > 0) by (service_name,set))
 >>>> max by (set) (max(max_over_time(mongodb_mongod_replset_member_replication_lag{cluster="$cluster"}[$interval]) > 0) by (instance,set) or max(max_over_time(mongodb_mongod_replset_member_replication_lag{cluster="$cluster"}[5m]) > 0) by (instance,set))

 <<<< max(max(mongodb_mongod_replset_oplog_head_timestamp{cluster="$cluster"}-mongodb_mongod_replset_oplog_tail_timestamp{cluster="$cluster"}) by (service_name) * on (service_name) group_right mongodb_mongod_replset_my_state{cluster="$cluster"} / mongodb_mongod_replset_my_state{cluster="$cluster"}) by (set)
 >>>> max(max(mongodb_mongod_replset_oplog_head_timestamp{cluster="$cluster"}-mongodb_mongod_replset_oplog_tail_timestamp{cluster="$cluster"}) by (instance) * on (instance) group_right mongodb_mongod_replset_my_state{cluster="$cluster"} / mongodb_mongod_replset_my_state{cluster="$cluster"}) by (set)

Done.