Powerwall-Dashboard
Powerwall-Dashboard copied to clipboard
Published
20 hours ago •
jasonacox
jasonacox
Telegraf Errors in new version
Hi-
The new version of Powerwall-Dashboard isn't working properly for me and telegraf has a bunch of these errors. Pypowerwall looks to be working correctly.
2022-04-19T14:29:30Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: invalid character 'E' looking for beginning of value
2022-04-19T14:29:35Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: invalid character 'E' looking for beginning of value
2022-04-19T14:29:35Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: invalid character 'E' looking for beginning of value
Hi @mcg88
That indicates that it is getting the "ERROR!" response from pypowerwall. What do you see when you browse over to these URLs (replace localhost with the address of the server running the dashboard):
http://localhost:8675/stats
This should show the version of pypowerwall running. It should be {"pypowerwall": "0.4.0 Proxy t8". If it is not, you can upgrade pypowerwall with this:
docker stop pypowerwall
docker rm pypowerwall
docker images | grep pypowerwall | awk '{print $3}' | xargs docker rmi -f
docker-compose -f powerwall.yml up -d
Also all of these (even if some error):
http://localhost:8675/freq http://localhost:8675/temps/pw http://localhost:8675/aggregates
If all of them error, it could also be that the credentials are wrong. Re-run the setup.sh and make sure the password is entered correctly.
Thanks for the response @jasonacox
Based on your recommendation, I found that pypowerwall was running 0.3.0 and I went ahead and upgraded it it. It is now on the correct version. Still having the same issue:
Stats endpoint is pasted below. Freq end point doesn't work. Temps and aggregates both appear to work.
{
"pypowerwall": "0.4.0 Proxy t8",
"gets": 80,
"errors": 28,
"timeout": 0,
"uri": {
"/stats": 25,
"/soe": 13,
"/strings": 13,
"/temps/pw": 14,
"/aggregates": 15
},
"ts": 1651000438,
"start": 1651000344,
"clear": 1651000344
}
Hi @mcg88 can you see if you can get to any of these (replace localhost with the IP address of the dashboard)?
http://localhost:8675/version
http://localhost:8675/vitals
I'm starting to suspect that the error for http://localhost:8675/freq could be due to some missing vital data that the proxy uses to get the frequencies.
You can also check for logs on the host:
docker logs pypowerwall
@jasonacox
I can access both vitals and version endpoints.
Logs:
Exception happened during processing of request from ('172.18.0.4', 56302)
Traceback (most recent call last):
File "/usr/local/lib/python3.7/socketserver.py", line 650, in process_request_thread
self.finish_request(request, client_address)
File "/usr/local/lib/python3.7/socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/local/lib/python3.7/socketserver.py", line 720, in __init__
self.handle()
File "/usr/local/lib/python3.7/http/server.py", line 426, in handle
self.handle_one_request()
File "/usr/local/lib/python3.7/http/server.py", line 414, in handle_one_request
method()
File "server.py", line 165, in do_GET
fcv["METER_X_VL1N"] = d['METER_X_VL1N']
KeyError: 'METER_X_VL1N'
@mcg88 Would you be able to paste part of your vitals? I'm looking for the details of the TESYNC section. I suspect your system is using something else to report voltages.
"TESYNC--1493315-01-F--x": {
"ISLAND_FreqL1_Load": 60.0,
"ISLAND_FreqL1_Main": 60.0,
"ISLAND_FreqL2_Load": 60.0,
"ISLAND_FreqL2_Main": 60.0,
"ISLAND_FreqL3_Load": 0.0,
"ISLAND_FreqL3_Main": 0.0,
"ISLAND_GridConnected": true,
"ISLAND_GridState": "ISLAND_GridState_Grid_Compliant",
"ISLAND_L1L2PhaseDelta": -256.0,
"ISLAND_L1L3PhaseDelta": -256.0,
"ISLAND_L1MicrogridOk": true,
"ISLAND_L2L3PhaseDelta": -256.0,
"ISLAND_L2MicrogridOk": true,
"ISLAND_L3MicrogridOk": false,
"ISLAND_PhaseL1_Main_Load": 0.0,
"ISLAND_PhaseL2_Main_Load": 0.0,
"ISLAND_PhaseL3_Main_Load": -256.0,
"ISLAND_ReadyForSynchronization": true,
"ISLAND_VL1N_Load": 120.5,
"ISLAND_VL1N_Main": 120.5,
"ISLAND_VL2N_Load": 120.5,
"ISLAND_VL2N_Main": 120.5,
"ISLAND_VL3N_Load": 0.0,
"ISLAND_VL3N_Main": 0.0,
"METER_X_CTA_I": 3.1485,
"METER_X_CTA_InstReactivePower": -140.0,
"METER_X_CTA_InstRealPower": 268.0,
"METER_X_CTB_I": 2.7720000000000002,
"METER_X_CTB_InstReactivePower": -90.0,
"METER_X_CTB_InstRealPower": -275.0,
"METER_X_CTC_I": 0.0,
"METER_X_CTC_InstReactivePower": 0.0,
"METER_X_CTC_InstRealPower": 0.0,
"METER_X_LifetimeEnergyExport": 1707894.0,
"METER_X_LifetimeEnergyImport": 3233722.0,
"METER_X_VL1N": 119.84,
"METER_X_VL2N": 120.11,
"METER_X_VL3N": 0.0,
"METER_Y_CTA_I": 0.0,
"METER_Y_CTA_InstReactivePower": 0.0,
"METER_Y_CTA_InstRealPower": 0.0,
"METER_Y_CTB_I": 0.0,
"METER_Y_CTB_InstReactivePower": 0.0,
"METER_Y_CTB_InstRealPower": 0.0,
"METER_Y_CTC_I": 0.0,
"METER_Y_CTC_InstReactivePower": 0.0,
"METER_Y_CTC_InstRealPower": 0.0,
"METER_Y_LifetimeEnergyExport": 0.0,
"METER_Y_LifetimeEnergyImport": 2.0,
"METER_Y_VL1N": 119.85000000000001,
"METER_Y_VL2N": 120.11,
"METER_Y_VL3N": 0.0,
"SYNC_ExternallyPowered": false,
"SYNC_SiteSwitchEnabled": false,
"alerts": [
"SYNC_a001_SW_App_Boot"
],
"componentParentDin": "STSTSM--1232100-00-E--x",
"firmwareVersion": "bce98cade591cb",
"lastCommunicationTime": 1651120300,
"manufacturer": "TESLA",
"partNumber": "1493315-01-F",
"serialNumber": "x",
"teslaEnergyEcuAttributes": {
"ecuType": 259
}
},
Here is the TESYNC section. To note- I've been using the Dashboard for a while without any problems until it suddenly stopped working and throwing these errors. No hardware changes and no software updates aside from what may be automatic / over the air
"TESYNC--1118431-00-L--x": {
"ISLAND_FreqL1_Load": 60,
"ISLAND_FreqL1_Main": 60,
"ISLAND_FreqL2_Load": 60.010000000000005,
"ISLAND_FreqL2_Main": 60.010000000000005,
"ISLAND_FreqL3_Load": 0,
"ISLAND_FreqL3_Main": 0,
"ISLAND_GridConnected": true,
"ISLAND_GridState": "ISLAND_GridState_Grid_Compliant",
"ISLAND_L1L2PhaseDelta": -256,
"ISLAND_L1L3PhaseDelta": -256,
"ISLAND_L1MicrogridOk": true,
"ISLAND_L2L3PhaseDelta": -256,
"ISLAND_L2MicrogridOk": true,
"ISLAND_L3MicrogridOk": false,
"ISLAND_PhaseL1_Main_Load": 0,
"ISLAND_PhaseL2_Main_Load": 0,
"ISLAND_PhaseL3_Main_Load": -256,
"ISLAND_ReadyForSynchronization": true,
"ISLAND_VL1N_Load": 119.5,
"ISLAND_VL1N_Main": 119.5,
"ISLAND_VL2N_Load": 118.5,
"ISLAND_VL2N_Main": 118.5,
"ISLAND_VL3N_Load": 0,
"ISLAND_VL3N_Main": 0,
"SYNC_ExternallyPowered": false,
"SYNC_SiteSwitchEnabled": true,
"componentParentDin": "STSTSM--1118431-00-L--x",
"firmwareVersion": "bce98cade591cb",
"lastCommunicationTime": 1651154738,
"manufacturer": "TESLA",
"partNumber": "1118431-00-L",
"serialNumber": "x",
"teslaEnergyEcuAttributes": {
"ecuType": 259
}
},
Thank you @mcg88 ! The issue is that your payload doesn't include the "METER" data points that were assumed to be there for the /freq macro in the proxy. I am going to adjust the proxy to ignore the missing data points and keep proceeding without error.
Just to clarify, you said the dashboard stopped working. The errors you are getting from pypowerwall should only impact the frequency and voltage graphs in the dashboard. The other graphs should still be working. Is that wrong?
I have updated pypowerwall and believe this will remove the errors you are seeing. You can upgrade using the upgrade.sh script or doing this manually:
# Delete pyPowerwall for Upgrade
docker stop pypowerwall
docker rm pypowerwall
docker images | grep pypowerwall | awk '{print $3}' | xargs docker rmi -f
# Restart Stack
docker-compose -f powerwall.yml up -d
Once upgraded you should see "pypowerwall": "0.4.0 Proxy t9" when you hit http://localhost:8675/stats.
@jasonacox
My stats page is still showing errors. Looking at logs, pypowerwall seems clean now, but telegraf still showing errors as seen in my first post here.
{
"pypowerwall": "0.4.0 Proxy t9",
"gets": 118,
"errors": 33,
"timeout": 0,
"uri": {
"/stats": 26,
"/soe": 17,
"/temps/pw": 17,
"/strings": 17,
"/freq": 24,
"/aggregates": 17
},
"ts": 1651500306,
"start": 1651500204,
"clear": 1651500204
}
Here is a screenshot of what my dashboard looks like. Note the top graph is populated but no subsequent graph is.
Thanks @mcg88 - Can you paste some of the telegraf errors you are seeing? It really seems like there are some InfluxDB issues as some of those graphs below are just down-sampling of the raw data showing up in the main graph. Check the Influxdb logs too.
Influxdb logs
ts=2022-05-03T04:10:01.121123Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE1byuG000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/daily/39 duration=59.266ms
ts=2022-05-03T04:10:01.121232Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE1byuG000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=59.373ms
ts=2022-05-03T04:10:01.266142Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE1bzhW000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T04:10:01.353006Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE1bzhW000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/monthly/6 duration=86.876ms
ts=2022-05-03T04:10:01.353067Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE1bzhW000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=86.935ms
ts=2022-05-03T04:10:04.276062Z lvl=info msg="TSM compaction (start)" log_id=0aDHAezW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0aE1cAT0000 op_name=tsm1_compact_group db_shard_id=36 op_event=start
ts=2022-05-03T04:10:04.276084Z lvl=info msg="Beginning compaction" log_id=0aDHAezW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0aE1cAT0000 op_name=tsm1_compact_group db_shard_id=36 tsm1_files_n=3
ts=2022-05-03T04:10:04.276088Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0aE1cAT0000 op_name=tsm1_compact_group db_shard_id=36 tsm1_index=0 tsm1_file=/var/lib/influxdb/data/powerwall/raw/36/000000001-000000001.tsm
ts=2022-05-03T04:10:04.276092Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0aE1cAT0000 op_name=tsm1_compact_group db_shard_id=36 tsm1_index=1 tsm1_file=/var/lib/influxdb/data/powerwall/raw/36/000000003-000000001.tsm
ts=2022-05-03T04:10:04.276095Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0aE1cAT0000 op_name=tsm1_compact_group db_shard_id=36 tsm1_index=2 tsm1_file=/var/lib/influxdb/data/powerwall/raw/36/000000004-000000001.tsm
ts=2022-05-03T04:10:04.834935Z lvl=info msg="Compacted file" log_id=0aDHAezW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0aE1cAT0000 op_name=tsm1_compact_group db_shard_id=36 tsm1_index=0 tsm1_file=/var/lib/influxdb/data/powerwall/raw/36/000000004-000000002.tsm.tmp
ts=2022-05-03T04:10:04.834975Z lvl=info msg="Finished compacting files" log_id=0aDHAezW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0aE1cAT0000 op_name=tsm1_compact_group db_shard_id=36 tsm1_files_n=1
ts=2022-05-03T04:10:04.834981Z lvl=info msg="TSM compaction (end)" log_id=0aDHAezW000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0aE1cAT0000 op_name=tsm1_compact_group db_shard_id=36 op_event=end op_elapsed=558.924ms
ts=2022-05-03T04:33:06.405200Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aE2wXPG000 op_name=retention_delete_check op_event=start
ts=2022-05-03T04:33:06.406512Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aE2wXPG000 op_name=retention_delete_check op_event=end op_elapsed=1.326ms
ts=2022-05-03T05:03:05.195171Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aE4eJvl000 op_name=retention_delete_check op_event=start
ts=2022-05-03T05:03:05.195388Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aE4eJvl000 op_name=retention_delete_check op_event=end op_elapsed=0.284ms
ts=2022-05-03T05:10:00.685134Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE52fwG000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T05:10:00.754709Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE52fwG000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/daily/39 duration=69.601ms
ts=2022-05-03T05:10:00.754910Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE52fwG000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=69.792ms
ts=2022-05-03T05:10:00.901015Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE52gmG000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T05:10:00.972859Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE52gmG000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/monthly/6 duration=71.858ms
ts=2022-05-03T05:10:00.972890Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE52gmG000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=71.887ms
ts=2022-05-03T05:10:01.907075Z lvl=info msg="TSM compaction (start)" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aE52khl000 op_name=tsm1_compact_group db_shard_id=6 op_event=start
ts=2022-05-03T05:10:01.907122Z lvl=info msg="Beginning compaction" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aE52khl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_files_n=8
ts=2022-05-03T05:10:01.907128Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aE52khl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=0 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000323-000000001.tsm
ts=2022-05-03T05:10:01.907132Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aE52khl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=1 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000324-000000001.tsm
ts=2022-05-03T05:10:01.907135Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aE52khl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=2 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000325-000000001.tsm
ts=2022-05-03T05:10:01.907137Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aE52khl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=3 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000326-000000001.tsm
ts=2022-05-03T05:10:01.907140Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aE52khl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=4 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000327-000000001.tsm
ts=2022-05-03T05:10:01.907143Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aE52khl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=5 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000328-000000001.tsm
ts=2022-05-03T05:10:01.907146Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aE52khl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=6 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000329-000000001.tsm
ts=2022-05-03T05:10:01.907149Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aE52khl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=7 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000330-000000001.tsm
ts=2022-05-03T05:10:02.149692Z lvl=info msg="Compacted file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aE52khl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=0 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000330-000000002.tsm.tmp
ts=2022-05-03T05:10:02.149723Z lvl=info msg="Finished compacting files" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aE52khl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_files_n=1
ts=2022-05-03T05:10:02.149731Z lvl=info msg="TSM compaction (end)" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aE52khl000 op_name=tsm1_compact_group db_shard_id=6 op_event=end op_elapsed=242.669ms
ts=2022-05-03T05:33:03.960881Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aE6N6M0000 op_name=retention_delete_check op_event=start
ts=2022-05-03T05:33:03.960953Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aE6N6M0000 op_name=retention_delete_check op_event=end op_elapsed=0.086ms
ts=2022-05-03T06:03:02.758787Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aE84tuW000 op_name=retention_delete_check op_event=start
ts=2022-05-03T06:03:02.759001Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aE84tuW000 op_name=retention_delete_check op_event=end op_elapsed=0.229ms
ts=2022-05-03T06:10:00.244669Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE8UNi0000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T06:10:00.327915Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE8UNi0000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/daily/39 duration=83.261ms
ts=2022-05-03T06:10:00.328070Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE8UNi0000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=83.414ms
ts=2022-05-03T06:10:00.457527Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE8UOYG000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T06:10:00.526907Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE8UOYG000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/monthly/6 duration=69.391ms
ts=2022-05-03T06:10:00.526985Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE8UOYG000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=69.467ms
ts=2022-05-03T06:15:00.694540Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE8liLW000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T06:15:00.965755Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE8liLW000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/_internal/monitor/43 duration=271.237ms
ts=2022-05-03T06:15:00.966076Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aE8liLW000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=271.551ms
ts=2022-05-03T06:33:01.499978Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aE9ngEl000 op_name=retention_delete_check op_event=start
ts=2022-05-03T06:33:01.500170Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aE9ngEl000 op_name=retention_delete_check op_event=end op_elapsed=0.333ms
ts=2022-05-03T07:03:00.342925Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aEBWTyW000 op_name=retention_delete_check op_event=start
ts=2022-05-03T07:03:00.342998Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aEBWTyW000 op_name=retention_delete_check op_event=end op_elapsed=0.125ms
ts=2022-05-03T07:10:00.843378Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEBv8Yl000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T07:10:00.964261Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEBv8Yl000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/daily/39 duration=120.903ms
ts=2022-05-03T07:10:00.964381Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEBv8Yl000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=121.020ms
ts=2022-05-03T07:10:01.053288Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEBv9NG000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T07:10:01.108293Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEBv9NG000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/monthly/6 duration=55.017ms
ts=2022-05-03T07:10:01.108356Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEBv9NG000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=55.079ms
ts=2022-05-03T07:10:01.843336Z lvl=info msg="TSM compaction (start)" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEBvCSl000 op_name=tsm1_compact_group db_shard_id=39 op_event=start
ts=2022-05-03T07:10:01.843383Z lvl=info msg="Beginning compaction" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEBvCSl000 op_name=tsm1_compact_group db_shard_id=39 tsm1_files_n=8
ts=2022-05-03T07:10:01.843392Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEBvCSl000 op_name=tsm1_compact_group db_shard_id=39 tsm1_index=0 tsm1_file=/var/lib/influxdb/data/powerwall/daily/39/000000018-000000001.tsm
ts=2022-05-03T07:10:01.843398Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEBvCSl000 op_name=tsm1_compact_group db_shard_id=39 tsm1_index=1 tsm1_file=/var/lib/influxdb/data/powerwall/daily/39/000000019-000000001.tsm
ts=2022-05-03T07:10:01.843403Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEBvCSl000 op_name=tsm1_compact_group db_shard_id=39 tsm1_index=2 tsm1_file=/var/lib/influxdb/data/powerwall/daily/39/000000020-000000001.tsm
ts=2022-05-03T07:10:01.843406Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEBvCSl000 op_name=tsm1_compact_group db_shard_id=39 tsm1_index=3 tsm1_file=/var/lib/influxdb/data/powerwall/daily/39/000000021-000000001.tsm
ts=2022-05-03T07:10:01.843409Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEBvCSl000 op_name=tsm1_compact_group db_shard_id=39 tsm1_index=4 tsm1_file=/var/lib/influxdb/data/powerwall/daily/39/000000022-000000001.tsm
ts=2022-05-03T07:10:01.843412Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEBvCSl000 op_name=tsm1_compact_group db_shard_id=39 tsm1_index=5 tsm1_file=/var/lib/influxdb/data/powerwall/daily/39/000000023-000000001.tsm
ts=2022-05-03T07:10:01.843415Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEBvCSl000 op_name=tsm1_compact_group db_shard_id=39 tsm1_index=6 tsm1_file=/var/lib/influxdb/data/powerwall/daily/39/000000024-000000001.tsm
ts=2022-05-03T07:10:01.843417Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEBvCSl000 op_name=tsm1_compact_group db_shard_id=39 tsm1_index=7 tsm1_file=/var/lib/influxdb/data/powerwall/daily/39/000000025-000000001.tsm
ts=2022-05-03T07:10:01.993490Z lvl=info msg="Compacted file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEBvCSl000 op_name=tsm1_compact_group db_shard_id=39 tsm1_index=0 tsm1_file=/var/lib/influxdb/data/powerwall/daily/39/000000025-000000002.tsm.tmp
ts=2022-05-03T07:10:01.993580Z lvl=info msg="Finished compacting files" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEBvCSl000 op_name=tsm1_compact_group db_shard_id=39 tsm1_files_n=1
ts=2022-05-03T07:10:01.993594Z lvl=info msg="TSM compaction (end)" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEBvCSl000 op_name=tsm1_compact_group db_shard_id=39 op_event=end op_elapsed=150.265ms
ts=2022-05-03T07:32:59.107370Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aEDEGOl000 op_name=retention_delete_check op_event=start
ts=2022-05-03T07:32:59.107958Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aEDEGOl000 op_name=retention_delete_check op_event=end op_elapsed=0.610ms
ts=2022-05-03T08:02:57.909064Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aEEx2yG000 op_name=retention_delete_check op_event=start
ts=2022-05-03T08:02:57.909134Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aEEx2yG000 op_name=retention_delete_check op_event=end op_elapsed=0.081ms
ts=2022-05-03T08:10:00.410094Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEFLqMW000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T08:10:00.464918Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEFLqMW000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/daily/39 duration=54.840ms
ts=2022-05-03T08:10:00.464979Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEFLqMW000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=54.898ms
ts=2022-05-03T08:10:00.616745Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEFLrA0000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T08:10:00.669486Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEFLrA0000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/monthly/6 duration=52.752ms
ts=2022-05-03T08:10:00.669947Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEFLrA0000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=53.210ms
ts=2022-05-03T08:32:56.702142Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aEGeqVW000 op_name=retention_delete_check op_event=start
ts=2022-05-03T08:32:56.703145Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aEGeqVW000 op_name=retention_delete_check op_event=end op_elapsed=1.024ms
ts=2022-05-03T09:02:55.472168Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aEINcx0000 op_name=retention_delete_check op_event=start
ts=2022-05-03T09:02:55.472483Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aEINcx0000 op_name=retention_delete_check op_event=end op_elapsed=0.330ms
ts=2022-05-03T09:09:59.966627Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEImY7W000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T09:10:00.033826Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEImY7W000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/daily/39 duration=67.233ms
ts=2022-05-03T09:10:00.033960Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEImY7W000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=67.365ms
ts=2022-05-03T09:10:00.180350Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEImYy0000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T09:10:00.280007Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEImYy0000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/monthly/6 duration=99.704ms
ts=2022-05-03T09:10:00.280357Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEImYy0000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=100.018ms
ts=2022-05-03T09:22:43.115277Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEJW7Al000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T09:22:43.462835Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEJW7Al000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/_internal/monitor/43 duration=347.582ms
ts=2022-05-03T09:22:43.462995Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEJW7Al000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=347.739ms
ts=2022-05-03T09:32:54.263951Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aEK5QTl000 op_name=retention_delete_check op_event=start
ts=2022-05-03T09:32:54.264102Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aEK5QTl000 op_name=retention_delete_check op_event=end op_elapsed=0.164ms
ts=2022-05-03T09:52:31.573132Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aELDHLG000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T09:52:31.985020Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aELDHLG000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/raw/42 duration=411.912ms
ts=2022-05-03T09:52:31.985084Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aELDHLG000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=411.974ms
ts=2022-05-03T10:02:53.015968Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aELoCql000 op_name=retention_delete_check op_event=start
ts=2022-05-03T10:02:53.017092Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aELoCql000 op_name=retention_delete_check op_event=end op_elapsed=1.140ms
ts=2022-05-03T10:10:00.536594Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEMDIr0000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T10:10:00.582108Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEMDIr0000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/daily/39 duration=45.538ms
ts=2022-05-03T10:10:00.582224Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEMDIr0000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=45.643ms
ts=2022-05-03T10:10:00.753625Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEMDJhG000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T10:10:00.809911Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEMDJhG000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/monthly/6 duration=56.360ms
ts=2022-05-03T10:10:00.809950Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEMDJhG000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=56.395ms
ts=2022-05-03T10:32:51.820389Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aENW~R0000 op_name=retention_delete_check op_event=start
ts=2022-05-03T10:32:51.820891Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aENW~R0000 op_name=retention_delete_check op_event=end op_elapsed=0.515ms
ts=2022-05-03T11:02:50.643338Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aEPEn4l000 op_name=retention_delete_check op_event=start
ts=2022-05-03T11:02:50.643580Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aEPEn4l000 op_name=retention_delete_check op_event=end op_elapsed=0.257ms
ts=2022-05-03T11:10:00.131626Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEPd~ll000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T11:10:00.175032Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEPd~ll000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/daily/39 duration=43.432ms
ts=2022-05-03T11:10:00.175069Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEPd~ll000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=43.458ms
ts=2022-05-03T11:10:00.345827Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEPe0bG000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T11:10:00.415048Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEPe0bG000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/monthly/6 duration=69.233ms
ts=2022-05-03T11:10:00.415111Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEPe0bG000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=69.293ms
ts=2022-05-03T11:32:49.394698Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aEQx_SW000 op_name=retention_delete_check op_event=start
ts=2022-05-03T11:32:49.395048Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aEQx_SW000 op_name=retention_delete_check op_event=end op_elapsed=0.371ms
ts=2022-05-03T12:02:48.194611Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aESfN0W000 op_name=retention_delete_check op_event=start
ts=2022-05-03T12:02:48.194766Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aESfN0W000 op_name=retention_delete_check op_event=end op_elapsed=0.272ms
ts=2022-05-03T12:10:00.673252Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aET4lOG000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T12:10:00.745073Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aET4lOG000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/daily/39 duration=71.843ms
ts=2022-05-03T12:10:00.745109Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aET4lOG000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=71.876ms
ts=2022-05-03T12:10:00.883471Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aET4mCl000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T12:10:00.961382Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aET4mCl000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/monthly/6 duration=77.926ms
ts=2022-05-03T12:10:00.961414Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aET4mCl000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=77.955ms
ts=2022-05-03T12:30:25.527465Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEUFWyl000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T12:30:25.834292Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEUFWyl000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/_internal/monitor/43 duration=306.834ms
ts=2022-05-03T12:30:25.834543Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEUFWyl000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=307.090ms
ts=2022-05-03T12:32:46.985487Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aEUO9YG000 op_name=retention_delete_check op_event=start
ts=2022-05-03T12:32:46.985571Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aEUO9YG000 op_name=retention_delete_check op_event=end op_elapsed=0.111ms
ts=2022-05-03T13:02:45.756386Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aEW5w~0000 op_name=retention_delete_check op_event=start
ts=2022-05-03T13:02:45.756701Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aEW5w~0000 op_name=retention_delete_check op_event=end op_elapsed=0.340ms
ts=2022-05-03T13:10:00.230331Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEWWT9W000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T13:10:00.286232Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEWWT9W000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/daily/39 duration=55.914ms
ts=2022-05-03T13:10:00.286680Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEWWT9W000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=56.358ms
ts=2022-05-03T13:10:00.450273Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEWWU0W000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T13:10:00.513004Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEWWU0W000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/monthly/6 duration=62.759ms
ts=2022-05-03T13:10:00.513042Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEWWU0W000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=62.794ms
ts=2022-05-03T13:10:01.443095Z lvl=info msg="TSM compaction (start)" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEWWXtl000 op_name=tsm1_compact_group db_shard_id=6 op_event=start
ts=2022-05-03T13:10:01.443119Z lvl=info msg="Beginning compaction" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEWWXtl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_files_n=8
ts=2022-05-03T13:10:01.443123Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEWWXtl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=0 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000331-000000001.tsm
ts=2022-05-03T13:10:01.443127Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEWWXtl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=1 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000332-000000001.tsm
ts=2022-05-03T13:10:01.443130Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEWWXtl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=2 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000333-000000001.tsm
ts=2022-05-03T13:10:01.443132Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEWWXtl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=3 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000334-000000001.tsm
ts=2022-05-03T13:10:01.443135Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEWWXtl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=4 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000335-000000001.tsm
ts=2022-05-03T13:10:01.443139Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEWWXtl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=5 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000336-000000001.tsm
ts=2022-05-03T13:10:01.443143Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEWWXtl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=6 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000337-000000001.tsm
ts=2022-05-03T13:10:01.443148Z lvl=info msg="Compacting file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEWWXtl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=7 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000338-000000001.tsm
ts=2022-05-03T13:10:01.696131Z lvl=info msg="Compacted file" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEWWXtl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_index=0 tsm1_file=/var/lib/influxdb/data/powerwall/monthly/6/000000338-000000002.tsm.tmp
ts=2022-05-03T13:10:01.696190Z lvl=info msg="Finished compacting files" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEWWXtl000 op_name=tsm1_compact_group db_shard_id=6 tsm1_files_n=1
ts=2022-05-03T13:10:01.696199Z lvl=info msg="TSM compaction (end)" log_id=0aDHAezW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0aEWWXtl000 op_name=tsm1_compact_group db_shard_id=6 op_event=end op_elapsed=253.107ms
ts=2022-05-03T13:32:44.548998Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aEXojX0000 op_name=retention_delete_check op_event=start
ts=2022-05-03T13:32:44.549142Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aEXojX0000 op_name=retention_delete_check op_event=end op_elapsed=0.163ms
ts=2022-05-03T14:02:43.304426Z lvl=info msg="Retention policy deletion check (start)" log_id=0aDHAezW000 service=retention trace_id=0aEZXWv0000 op_name=retention_delete_check op_event=start
ts=2022-05-03T14:02:43.304562Z lvl=info msg="Retention policy deletion check (end)" log_id=0aDHAezW000 service=retention trace_id=0aEZXWv0000 op_name=retention_delete_check op_event=end op_elapsed=0.209ms
ts=2022-05-03T14:02:43.306736Z lvl=info msg="Sending usage statistics to usage.influxdata.com" log_id=0aDHAezW000
ts=2022-05-03T14:09:59.825838Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEZxA4G000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T14:09:59.872700Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEZxA4G000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/daily/39 duration=46.904ms
ts=2022-05-03T14:09:59.872737Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEZxA4G000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=46.938ms
ts=2022-05-03T14:10:00.030447Z lvl=info msg="Cache snapshot (start)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEZxAsW000 op_name=tsm1_cache_snapshot op_event=start
ts=2022-05-03T14:10:00.106134Z lvl=info msg="Snapshot for path written" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEZxAsW000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/powerwall/monthly/6 duration=75.698ms
ts=2022-05-03T14:10:00.106495Z lvl=info msg="Cache snapshot (end)" log_id=0aDHAezW000 engine=tsm1 trace_id=0aEZxAsW000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=76.056ms
Telegraf
2022-05-02T14:02:56Z I! [agent] Hang on, flushing any cached metrics before shutdown
2022-05-02T14:02:56Z I! [agent] Stopping running outputs
2022-05-02T14:03:43Z I! Starting Telegraf 1.21.2
2022-05-02T14:03:43Z I! Using config file: /etc/telegraf/telegraf.conf
2022-05-02T14:03:43Z I! Loaded inputs: cpu disk diskio http kernel mem processes swap system
2022-05-02T14:03:43Z I! Loaded aggregators:
2022-05-02T14:03:43Z I! Loaded processors: date (2x)
2022-05-02T14:03:43Z I! Loaded outputs: influxdb
2022-05-02T14:03:43Z I! Tags enabled: host=telegraf
2022-05-02T14:03:43Z I! [agent] Config: Interval:5s, Quiet:false, Hostname:"telegraf", Flush Interval:10s
2022-05-02T14:22:38Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-02T14:22:38Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-02T18:06:44Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-02T18:12:33Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-02T18:12:33Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-02T18:12:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-02T18:12:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-02T18:12:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-02T18:12:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-02T18:12:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-02T18:51:24Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-02T20:00:19Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-02T20:00:19Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-02T20:00:20Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": EOF
2022-05-02T20:00:20Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": EOF
2022-05-02T20:00:20Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": EOF
2022-05-02T20:00:20Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": EOF
2022-05-02T20:00:20Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": EOF
2022-05-02T20:00:25Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": EOF
2022-05-02T20:00:25Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": EOF
2022-05-02T20:00:25Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": EOF
2022-05-02T20:00:25Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": EOF
2022-05-02T20:00:25Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": EOF
2022-05-02T20:00:30Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": EOF
2022-05-02T20:00:30Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": EOF
2022-05-02T20:00:30Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": EOF
2022-05-02T20:00:30Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": EOF
2022-05-02T20:00:30Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": EOF
2022-05-02T20:00:35Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": EOF
2022-05-02T20:00:35Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": EOF
2022-05-02T20:00:35Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": EOF
2022-05-02T20:00:35Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": EOF
2022-05-02T20:00:35Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": EOF
2022-05-02T20:18:33Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-02T21:14:58Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-02T23:06:04Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-02T23:07:04Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T00:58:23Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T00:59:14Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T00:59:14Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T00:59:14Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T00:59:14Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T00:59:14Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T00:59:19Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T00:59:19Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T00:59:19Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T00:59:19Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T00:59:19Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T00:59:20Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": EOF
2022-05-03T00:59:20Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": EOF
2022-05-03T00:59:20Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": EOF
2022-05-03T00:59:20Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": EOF
2022-05-03T00:59:20Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": EOF
2022-05-03T01:18:09Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T01:18:09Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T01:18:09Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T01:18:09Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T01:18:09Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:25:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:26:19Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:26:39Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:26:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:26:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:26:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:26:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:26:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:27:13Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:27:13Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:27:13Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:27:34Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:27:39Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:27:39Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:27:48Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:27:48Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:36:48Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:36:48Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:36:48Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:36:48Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:36:48Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:36:54Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:36:54Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:36:54Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:36:54Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:36:54Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:36:55Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": EOF
2022-05-03T04:36:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:36:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:36:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:36:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:46:04Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:46:04Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:47:18Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T04:47:18Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T08:16:04Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T08:16:04Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T08:16:04Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T11:07:54Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T11:07:54Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T11:07:54Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T11:07:54Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T11:22:38Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T14:11:39Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T14:12:09Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T14:12:09Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T14:15:03Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T14:15:03Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T14:15:28Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-05-03T14:15:28Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Hi @mcg88 - Thanks for these! The errors in telegraf and what you see in the grafana dashboard has me puzzled. Was it working before the upgrade to the new version?
If you can, I would recommend setting up a clean install on another system (if you have one). Otherwise, you can archive and try a new build on your current host. Your installation may vary, but I would do something like this:
# Clean up old containers and file system
echo "Stopping and Cleaning up old containers..."
docker-compose -f powerwall.yml down
docker-compose -f powerwall.yml rm
docker rm pypowerwall
docker images | grep pypowerwall | awk '{print $3}' | xargs docker rmi -f
docker rm telegraf
docker images | grep telegraf | awk '{print $3}' | xargs docker rmi -f
docker rm grafana
docker images | grep grafana | awk '{print $3}' | xargs docker rmi -f
docker rm influxdb
docker images | grep influxdb | awk '{print $3}' | xargs docker rmi -f
# Archive Old Setup
cd ..
mv Powerwall-Dashboard Powerwall-Dashboard.old
# Pull New Install
echo "Cloning from github..."
git clone https://github.com/jasonacox/Powerwall-Dashboard.git
cd Powerwall-Dashboard
# Setup
echo "Starting setup..."
./setup.sh
Closing this for now - feel free to re-open if the issue persists and we can troubleshoot further.