systemd problem ?

[pyDraco9]

when i use systemd to startup, it can't get RPC data, and nomore wrong log. and just "sudo systemd restart chia-monitor" will return to normal.

i don't know why, do you have any clues?

[philippnormann]

Maybe your chia-monitor service is launched before the chia daemons have fully started, and that's why you get connection errors at startup?

[philippnormann]

If you are also using systemd to launch your chia daemons, this might be helpful: https://serverfault.com/questions/482730/systemd-dependencies-and-boot-order

[giannoug]

Same here, more specifically the chia_sync_status gauge doesn't update.

I have both chia and chia-monitor as systemd services. This is how I've setup the chia-monitor service:

[Unit]
Description=chia-monitor prometheus exporter
Wants=network-online.target chia.service
After=network.target network-online.target chia.service
Requires=chia.service
BindsTo=chia.service
StartLimitIntervalSec=0

[Service]
Restart=always
User=chiauser
ExecStart=/usr/bin/env bash -c 'cd ~/chia-monitor; pipenv run python -m monitor'

[Install]
WantedBy=multi-user.target

On boot, chia-monitor chia_sync_status gauge reports constantly a value of 0.0. After restarting the service it goes back to 1.0.

[sorz]

Systemd doesn't know when chia is FULLY started unless chia tell it explicitly (via sd_notify).

It would be much easier that chia-monitor can tolerate such errors, once chia started fully, chia-monitor start working normally.

[sorz]

I just talk with @yagamiraku privately, he say that there was no error message on chia-monitor side, everything looks normal except gauges didn't get update, like @giannoug said.

[pyDraco9]

On boot, it like this:

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 14554.0
python_gc_objects_collected_total{generation="1"} 1692.0
python_gc_objects_collected_total{generation="2"} 300.0
# HELP python_gc_objects_uncollectable_total Uncollectable object found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 20021.0
python_gc_collections_total{generation="1"} 1820.0
python_gc_collections_total{generation="2"} 160.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="8",patchlevel="10",version="3.8.10"} 1.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 6.67844608e+08
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.08560384e+08
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.62727941921e+09
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 87.91
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 15.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024.0
# HELP chia_confirmed_total_mojos Sum of confirmed wallet balances
# TYPE chia_confirmed_total_mojos gauge
chia_confirmed_total_mojos 0.0
# HELP chia_farmed_total_mojos Total chia farmed
# TYPE chia_farmed_total_mojos gauge
chia_farmed_total_mojos 0.0
# HELP chia_network_space Approximation of current netspace
# TYPE chia_network_space gauge
chia_network_space 0.0
# HELP chia_diffculty Current networks farming difficulty
# TYPE chia_diffculty gauge
chia_diffculty 0.0
# HELP chia_peak_height Block height of the current peak
# TYPE chia_peak_height gauge
chia_peak_height 0.0
# HELP chia_sync_status Sync status of the connected full node
# TYPE chia_sync_status gauge
chia_sync_status 0.0
# HELP chia_connections_count Count of peers that the node is currently connected to
# TYPE chia_connections_count gauge
# HELP chia_plot_count OG plot count being farmed by harvester
# TYPE chia_plot_count gauge
chia_plot_count{host="127.0.0.1"} 0.0
# HELP chia_plot_size Size of OG plots being farmed by harvester
# TYPE chia_plot_size gauge
chia_plot_size{host="127.0.0.1"} 0.0
# HELP chia_portable_plot_count Portable plot count being farmed by harvester
# TYPE chia_portable_plot_count gauge
chia_portable_plot_count{host="127.0.0.1"} 155.0
# HELP chia_portable_plot_size Size of portable plots being farmed by harvester
# TYPE chia_portable_plot_size gauge
chia_portable_plot_size{host="127.0.0.1"} 1.6867874191296e+013
# HELP chia_signage_points_total Received signage points
# TYPE chia_signage_points_total counter
chia_signage_points_total 20.0
# HELP chia_signage_points_created Received signage points
# TYPE chia_signage_points_created gauge
chia_signage_points_created 1.6272794260254927e+09
# HELP chia_signage_point_index Received signage point index
# TYPE chia_signage_point_index gauge
chia_signage_point_index 61.0
# HELP chia_block_challenges_total Attempted block challenges
# TYPE chia_block_challenges_total counter
chia_block_challenges_total 20.0
# HELP chia_block_challenges_created Attempted block challenges
# TYPE chia_block_challenges_created gauge
chia_block_challenges_created 1.6272794260255792e+09
# HELP chia_plots_passed_filter_total Plots passed filter
# TYPE chia_plots_passed_filter_total counter
chia_plots_passed_filter_total 5.0
# HELP chia_plots_passed_filter_created Plots passed filter
# TYPE chia_plots_passed_filter_created gauge
chia_plots_passed_filter_created 1.6272794260256197e+09
# HELP chia_proofs_found_total Proofs found
# TYPE chia_proofs_found_total counter
chia_proofs_found_total 0.0
# HELP chia_proofs_found_created Proofs found
# TYPE chia_proofs_found_created gauge
chia_proofs_found_created 1.627279426025661e+09
# HELP chia_current_pool_points Number of pooling points you have collected during this round
# TYPE chia_current_pool_points gauge
chia_current_pool_points{p2="xxxxxxxx",url="https://asia1.pool.space"} 90.0
# HELP chia_current_pool_difficulty Difficulty of partials you are submitting
# TYPE chia_current_pool_difficulty gauge
chia_current_pool_difficulty{p2="xxxxxxxx",url="https://asia1.pool.space"} 3.0
# HELP chia_pool_points_found_since_start Total number of pooling points found
# TYPE chia_pool_points_found_since_start gauge
chia_pool_points_found_since_start{p2="xxxxxxxx",url="https://asia1.pool.space"} 0.0
# HELP chia_pool_points_acknowledged_since_start Total number of pooling points acknowledged
# TYPE chia_pool_points_acknowledged_since_start gauge
chia_pool_points_acknowledged_since_start{p2="xxxxxxxx",url="https://asia1.pool.space"} 0.0
# HELP chia_pool_points_found_24h Number of pooling points found the last 24h
# TYPE chia_pool_points_found_24h gauge
chia_pool_points_found_24h{p2="xxxxxxxx",url="https://asia1.pool.space"} 0.0
# HELP chia_pool_points_acknowledged_24h Number of pooling points acknowledged the last 24h
# TYPE chia_pool_points_acknowledged_24h gauge
chia_pool_points_acknowledged_24h{p2="xxxxxxxx",url="https://asia1.pool.space"} 0.0
# HELP chia_num_pool_errors_24h Number of pool errors during the last 24 hours
# TYPE chia_num_pool_errors_24h gauge
chia_num_pool_errors_24h{p2="xxxxxxxx",url="https://asia1.pool.space"} 0.0
# HELP chia_price_usd_cent Current Chia price in USD cent
# TYPE chia_price_usd_cent gauge
chia_price_usd_cent 19444.0
# HELP chia_price_eur_cent Current Chia price in EUR cent
# TYPE chia_price_eur_cent gauge
chia_price_eur_cent 16501.0
# HELP chia_price_btc_satoshi Current Chia price in BTC satoshi
# TYPE chia_price_btc_satoshi gauge
chia_price_btc_satoshi 504318.0
# HELP chia_price_eth_gwei Current Chia price in ETH gwei
# TYPE chia_price_eth_gwei gauge
chia_price_eth_gwei 8.27059e+07

All zero about chia network.

[ruant]

Add a delay on the chia-farm service, to let chia spin everything up before the service is reported as running. ExecStartPost=/bin/sleep 30

Then the chia-monitor most likely will have a fully operational chia to talk to when it starts.

[giannoug]

@ruant Tried that on my chia-monitor service (since that service has issues and not chia). I actually added ExecStartPre=/bin/sleep 60 since 30s didn't fix it for me. This is a silly fix that adds holes to the dashboards, but it's better than manually having to restart the service

[ruant]

Yeah, def. not a proper fix, but a shitty workaround.