prometheus_rabbitmq_exporter
prometheus_rabbitmq_exporter copied to clipboard
Published
20 hours ago •
deadtrickster
deadtrickster
Metric filtering
Hello, I would like to filter out all queue metric. And keep just some basic cluster health etc..
I'm expecting:
- faster scrape time,
- reducing cpu usage while scraping
- and no filtering rules in the Prometheus configuration
With basic configuration (no filtering) i get 85364 rows! (scrape duration ~10sec) With the following config i got 51801 rows! (scrape duration ~8sec) I'ts still too much...
{prometheus, [
{rabbitmq_exporter, [
{path, "/metrics"},
{connections_total_enabled, true},
{memory_stat_enabled,false},
{queue_messages_stat, [{messages_published_total,false},
{messages_confirmed_total,false},
{messages_delivered_total,false},
{messages_delivered_no_ack_total,false},
{messages_get_total,false},
{messages_get_no_ack_total,false},
{messages_deliver_get_total,false},
{messages_redelivered_total,false},
{messages_returned_total,false}]},
{exchange_messages_stat, [{messages_published_total,false},
{messages_published_in_total,false},
{messages_published_out_total,false},
{messages_confirmed_total,false},
{messages_delivered_total,false},
{messages_delivered_no_ack_total,false},
{messages_get_total,false},
{messages_get_no_ack_total,false},
{messages_deliver_get_total,false},
{messages_redelivered_total,false},
{messages_returned_total,false}]}
]}
Metric counts summary:
awk -F{ '{print $1}' /rb-plugin_SHORT | grep -v '^#' | sort | uniq -c | sort -h
1
1 erlang_mnesia_committed_transactions 4713
1 erlang_mnesia_failed_transactions 2
1 erlang_mnesia_held_locks 0
1 erlang_mnesia_lock_queue 0
1 erlang_mnesia_logged_transactions 5
1 erlang_mnesia_restarted_transactions 765
1 erlang_mnesia_transaction_coordinators 0
1 erlang_mnesia_transaction_participants 0
1 erlang_vm_atom_count 38404
1 erlang_vm_atom_limit 5000000
1 erlang_vm_dirty_cpu_schedulers 24
1 erlang_vm_dirty_cpu_schedulers_online 24
1 erlang_vm_dirty_io_schedulers 10
1 erlang_vm_ets_limit 50000
1 erlang_vm_logical_processors 24
1 erlang_vm_logical_processors_available 24
1 erlang_vm_logical_processors_online 24
1 erlang_vm_memory_dets_tables 1
1 erlang_vm_memory_ets_tables 177
1 erlang_vm_port_count 18
1 erlang_vm_port_limit 65536
1 erlang_vm_process_count 7446
1 erlang_vm_process_limit 1048576
1 erlang_vm_schedulers 24
1 erlang_vm_schedulers_online 24
1 erlang_vm_smp_support 1
1 erlang_vm_statistics_bytes_output_total 82095010
1 erlang_vm_statistics_bytes_received_total 89728826
1 erlang_vm_statistics_context_switches 6522008
1 erlang_vm_statistics_garbage_collection_bytes_reclaimed 12941869408
1 erlang_vm_statistics_garbage_collection_number_of_gcs 5478847
1 erlang_vm_statistics_garbage_collection_words_reclaimed 1617733676
1 erlang_vm_statistics_reductions_total 1374252069
1 erlang_vm_statistics_run_queues_length_total 0
1 erlang_vm_statistics_runtime_milliseconds 543432
1 erlang_vm_statistics_wallclock_time_milliseconds 573721
1 erlang_vm_thread_pool_size 384
1 erlang_vm_threads 1
1 erlang_vm_time_correction 1
1 rabbitmq_channels 3813
1 rabbitmq_connections 3813
1 rabbitmq_connections_total
1 rabbitmq_consumers 3975
1 rabbitmq_exchanges 1143
1 rabbitmq_messages_confirmed_total 1742937
1 rabbitmq_messages_delivered_no_ack_total 0
1 rabbitmq_messages_delivered_total 1588108
1 rabbitmq_messages_deliver_get_total 1588108
1 rabbitmq_messages_get_no_ack_total 0
1 rabbitmq_messages_get_total 0
1 rabbitmq_messages_published_in_total NaN
1 rabbitmq_messages_published_out_total NaN
1 rabbitmq_messages_published_total 1743471
1 rabbitmq_messages_ready 0
1 rabbitmq_messages_redelivered_total 197
1 rabbitmq_messages_returned_total 0
1 rabbitmq_messages_unacknowledged 0
1 rabbitmq_queues 2332
1 rabbitmq_queues_disk_reads 0
1 rabbitmq_queues_disk_writes 0
1 telemetry_scrape_duration_seconds_count
1 telemetry_scrape_duration_seconds_sum
1 telemetry_scrape_size_bytes_count
1 telemetry_scrape_size_bytes_sum
2 erlang_vm_memory_atom_bytes_total
2 erlang_vm_memory_bytes_total
2 erlang_vm_memory_processes_bytes_total
2 telemetry_scrape_encoded_size_bytes_count
2 telemetry_scrape_encoded_size_bytes_sum
5 erlang_vm_memory_system_bytes_total
5 rabbitmq_node_up
17 rabbitmq_mnesia_table_disc_copies
17 rabbitmq_mnesia_table_disc_only_copies
17 rabbitmq_mnesia_table_disk_size_bytes
17 rabbitmq_mnesia_table_local_content
17 rabbitmq_mnesia_table_majority_required
17 rabbitmq_mnesia_table_master_nodes
17 rabbitmq_mnesia_table_memory_bytes
17 rabbitmq_mnesia_table_ram_copies
17 rabbitmq_mnesia_table_read_only
17 rabbitmq_mnesia_table_records_count
2332 rabbitmq_queue_auto_delete
2332 rabbitmq_queue_consumers
2332 rabbitmq_queue_consumer_utilisation
2332 rabbitmq_queue_disk_size_bytes
2332 rabbitmq_queue_durable
2332 rabbitmq_queue_exclusive
2332 rabbitmq_queue_head_message_timestamp
2332 rabbitmq_queue_memory
2332 rabbitmq_queue_message_bytes
2332 rabbitmq_queue_message_bytes_persistent
2332 rabbitmq_queue_message_bytes_ram
2332 rabbitmq_queue_message_bytes_ready
2332 rabbitmq_queue_message_bytes_unacknowledged
2332 rabbitmq_queue_messages
2332 rabbitmq_queue_messages_persistent
2332 rabbitmq_queue_messages_ram
2332 rabbitmq_queue_messages_ready
2332 rabbitmq_queue_messages_ready_ram
2332 rabbitmq_queue_messages_unacknowledged
2332 rabbitmq_queue_messages_unacknowledged_ram
2332 rabbitmq_queue_state
2576 erlang_vm_allocators
Hi, this was discussed here: https://github.com/deadtrickster/prometheus_rabbitmq_exporter/issues/18.
In short, just create your own collectors list and do not include queues_collector.
Hi, im still unable to remove queue metrics... Configuration(below) exports 103790 metrics! And scrape time is about 20sec.
Conffile:
1 [
2 {rabbit, [
3 {tcp_listeners, []},
4 {ssl_listeners, [5671,5672]},
5 {ssl_options, [{cacertfile,"/xxx/ca.pem"},
6 {certfile,"/xxx/cert"},
7 {keyfile,"/xxx/key"},
8 {verify,verify_peer},
9 {fail_if_no_peer_cert,false}]}
10 ]},
11 {prometheus, [
12 {rabbitmq_exporter, [
13 {path, "/metrics"},
14 {connections_total_enabled, false},
15 {queue_messages_stat, []},
16 {exchange_messages_stat, []}
17 ]},
18 {collectors, [
19
20 %% Standard prometheus collectors
21 prometheus_vm_statistics_collector,
22 prometheus_vm_system_info_collector,
23 prometheus_vm_memory_collector,
24 prometheus_mnesia_collector,
25
26 %% Rabbitmq collectors
27 prometheus_rabbitmq_overview_collector,
28 prometheus_rabbitmq_exchanges_collector,
29 prometheus_rabbitmq_mnesia_tables_collector,
30 prometheus_rabbitmq_nodes_collector
31 %%prometheus_rabbitmq_queues_collector
32 ]}
33 ]}
34 ].
Metric counts summary:
1518 rabbitmq_exchange_messages_confirmed_total
1518 rabbitmq_exchange_messages_delivered_no_ack_total
1518 rabbitmq_exchange_messages_delivered_total
1518 rabbitmq_exchange_messages_deliver_get_total
1518 rabbitmq_exchange_messages_get_no_ack_total
1518 rabbitmq_exchange_messages_get_total
1518 rabbitmq_exchange_messages_published_in_total
1518 rabbitmq_exchange_messages_published_out_total
1518 rabbitmq_exchange_messages_published_total
1518 rabbitmq_exchange_messages_redelivered_total
1518 rabbitmq_exchange_messages_returned_total
2576 erlang_vm_allocators
2809 rabbitmq_queue_auto_delete
2809 rabbitmq_queue_consumers
2809 rabbitmq_queue_consumer_utilisation
2809 rabbitmq_queue_disk_size_bytes
2809 rabbitmq_queue_durable
2809 rabbitmq_queue_exclusive
2809 rabbitmq_queue_head_message_timestamp
2809 rabbitmq_queue_memory
2809 rabbitmq_queue_message_bytes
2809 rabbitmq_queue_message_bytes_persistent
2809 rabbitmq_queue_message_bytes_ram
2809 rabbitmq_queue_message_bytes_ready
2809 rabbitmq_queue_message_bytes_unacknowledged
2809 rabbitmq_queue_messages
2809 rabbitmq_queue_messages_confirmed_total
2809 rabbitmq_queue_messages_delivered_no_ack_total
2809 rabbitmq_queue_messages_delivered_total
2809 rabbitmq_queue_messages_deliver_get_total
2809 rabbitmq_queue_messages_get_no_ack_total
2809 rabbitmq_queue_messages_get_total
2809 rabbitmq_queue_messages_persistent
2809 rabbitmq_queue_messages_published_total
2809 rabbitmq_queue_messages_ram
2809 rabbitmq_queue_messages_ready
2809 rabbitmq_queue_messages_ready_ram
2809 rabbitmq_queue_messages_redelivered_total
2809 rabbitmq_queue_messages_returned_total
2809 rabbitmq_queue_messages_unacknowledged
2809 rabbitmq_queue_messages_unacknowledged_ram
2809 rabbitmq_queue_state
@sznmaran Could you check that RabbitMQ is loading the configuration that you are trying to use?
You can use the rabbitmqctl environment command for that.
If I configure
{prometheus,
[{collectors,
[prometheus_vm_statistics_collector,
prometheus_vm_system_info_collector,prometheus_vm_memory_collector,
prometheus_mnesia_collector,prometheus_rabbitmq_overview_collector,
prometheus_rabbitmq_exchanges_collector,
prometheus_rabbitmq_mnesia_tables_collector,
prometheus_rabbitmq_nodes_collector]},
{rabbitmq_exporter,
[{queue_messages_stat,[]},{exchange_messages_stat,[]}]}]},
with 5000 queues, I get 1300 lines in the scrape and less than 1 second. With the default config, it takes 3 seconds and returns 151000 lines.
