Enhancement: Temperature line on Live Monitoring graph

[mp09]

Enhancement: Temperature line on Live Monitoring graph

Temperature is extremely relevant to power usage if you have/use Air Conditioning. It would be great to see a line graph of that on the live monitoring graph as another data input.

[mp09]

Seems like the lat/long info could be used to determine this as well (with a weather data source.

[jasonacox]

Hi @mp09 - That's definitely possible. The challenge we run into is that the API endpoints we scrape to get the local temperature based on lat/long tend to require that the user set up a API token and observe rate limiting on the amount of calls allowed. Grafana may have a plugin that helps. I'll see if I can figure out a good way to do this more generically, but would love some help.

[jasonacox]

I have an idea. There is a plugin for telegraf to pull OpenWeatherMap current location weather data (including temperature) - see https://github.com/influxdata/telegraf/blob/master/plugins/inputs/openweathermap/README.md.

• We could make this an option during setup (e.g. - Do you want to pull current weather data from OpenWeatherMap?)
• It would require the user to go to https://openweathermap.org/ to set up an account and copy the API key and city_id which they would paste into the setup process.
• If weather data is available, the dashboard.json can render it. Would the preference be to have it show up on the main Live Monitoring -> "Energy Usage" graph or a separate panel?

[jasonacox]

Ok, I used the OpenWeatherMap plugin to get the data into InfluxDB and have added it to the "Energy Usage" graph. Here is an example of adding Temperature (F) and Cloud Coverage (%). Not prefect but I plan to experiment with better visuals but looking to see if this is the general direction that would be helpful.

[jasonacox]

Update: I have created a branch called "weather" that is my running test configuration. Go here if you want to check it out. Keep in mind that it isn't integrated into setup.sh or upgrade.sh yet. I'm still testing the new container.

Here is my approach:

• I found the telegraf OpenWeatherMap integration to be glitchy (seemed to only pick up data every hour) and pivoted away from that. It may still work for some but I wanted something more stable and granular.
• I created a new service called weather411 that runs in a container and periodically (every 10m) reaches out to OpenWeatherMap.org to get the latest local weather data.
• Configuration: Will require users to sign up for a free account at OpenWeatherMap.org and get an API KEY.
• The API KEY, your Latitude and Longitude are entered into the weather411.conf config file. Ultimately this will be created by setup.sh or upgrade.sh.
• The weather411 service stores the data in InfluxDB in the powerwall DB allowing us to pull this into the existing Powerwall Dashboard.

Example overlay:

This only show the "temperature" and "clouds" (a % of cloudliness) data but there are many others available that may be interesting - I'm thinking:

• uvi (UV radiation - would be interesting to see if there was a correlation to production)
• rain (Nice to know when we get a free panel wash)
• wind (Potential cooling for higher efficiency)
• visibility (Takes into account more than just clouds but inverse metric)

Example weather data available from weather411 that I'm graphing:

[jasonacox]

BETA Testers

...or should I say, "Storm chasers"? 😄

If you want to try the weather data feature, here is how you can check it out.

Highly recommended: back up your data first.

# from the Powerwall-Dashboard folder

# First pull down the latest code from Github
git pull

# Second, switch to the weather branch (the BETA code)
git checkout weather

# Third, run the weather setup script and follow instructions
./weather.sh

OpenWeatherMap Note

After you sign up for the free OpenWeatherMap.org account, make sure you check your email and verify. Also, the API Key you get may not be active right away. It could take up to 2 hours but in my recent test it was working within 10 minutes. Please let me know your experience!

Dashboard Update

You need to install the newer dashboard.json that includes the weather data for the "live monitoring" graphs. In Grafana:

• From Dashboard\Manage (or Dashboard\Browse), select Import
• Upload one the dashboard.json in your Powerwall-Dashboard folder.
• If you want a weather dashboard, you can upload the test dashboard dashboard-weather.json in the weather/ folder.

Hopefully you will soon see something like this:

Issues

• The temperature data on the Live monitoring graph shows up as (%) instead of °C or °F. From what I can tell, Grafana only allows two axis. We have "kW" and "%". I put temperature on the "%" axis but it is confusing. If anyone has a better idea, please let me know.
• The graphs are basically set for °C but during the weather setup (weather.sh) you can choose Imperial units (°F) which is what will show up on the live monitoring instead of °C. However, for the weather dashboard, the values will be wrong as the axis is set for °C. Edit the panel and change the units from °C to °F.
• The weather411 service is storing all of the data from OpenWeatherMap into InfluxDB so there are many other data points you can play with. The weather411 graphs above show some of it.

How's the weather out there? 😁

Before I release this as v2.5.0 officially, I am running test and looking for feedback from the community. Please post any issues, questions and suggestions here.

[mp09]

awesome! thanks for working on this :)

Two issues so far. minor one, the /location.html site has latitude twice in the text

Second, it failed to find my file: Running Docker-Compose... ERROR: Couldn't find env file: /volume2/docker/grafana/Powerwall-Dashboard/grafana.env Weather Setup Complete

I'm not sure why that file isn't there though. things seem to be working fine.

[madkiwinz]

I didn't receive any install errors, but I needed to follow your earlier instructions to remove the weather docker image and force the docker image download. It seems to be working well.

[jasonacox]

minor one, the /location.html site has latitude twice in the text

Great catch!

Second, it failed to find my file: Running Docker-Compose... ERROR: Couldn't find env file: /volume2/docker/grafana/Powerwall-Dashboard/grafana.env Weather Setup Complete

Thanks @mp09 - This indicates you have an older install before we externalized the Grafana settings for users who wanted to customize it more. You can fix this by doing this:

cp grafana.env.sample grafana.env

Thanks @leemj001 - If you want to remove and re-install the latest weather docker image you can do this:

docker stop weather411
docker rm weather411
docker images | grep weather411 | awk '{print $3}' | xargs docker rmi -f
docker-compose -f powerwall.yml up -d

Thanks for your help and feedback!

[mp09]

That helped, thanks! However since deploying the newer version, it looks like grafana is unhappy with permissions -- though I'm not sure how permissions would've changed on the database: t=2022-08-21T17:37:55+0000 lvl=eror msg="Failed to rotate token" logger=context error="attempt to write a readonly database" t=2022-08-21T17:37:55+0000 lvl=eror msg="Failed to update last_seen_at" logger=context userId=1 orgId=1 uname=admin error="attempt to write a readonly database" t=2022-08-21T17:37:56+0000 lvl=eror msg="Failed to import dashboard" logger=context userId=1 orgId=1 uname=admin error="attempt to write a readonly database" t=2022-08-21T17:37:56+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/dashboards/import status=500 remote_addr=172.18.0.1 time_ms=139 size=40 referer=http://synology.local:9000/dashboard/import

but now I can't import the new dashboard .json

[jasonacox]

Interesting. Thanks @mp09 can you run a few commands to see what permissions you have on those files?

# what permission are set on the files and folders?
ls -la grafana

# try restart grafana?
docker restart grafana
docker logs grafana

Worse case, since Grafana is just the UI on the stack, you could completely remove grafana and start from a clean install (your powerwall data is in InfluxDB). Of course, I recommend a backup (e.g. sudo tar -zcvf backup.tgz Powerwall-Dashboard) 😁

# remove grafana container
docker stop grafana
docker rm grafana
docker images | grep grafana | awk '{print $3}' | xargs docker rmi -f

# rebuild grafana directory
mv grafana grafana.bak
mkdir grafana
docker-compose -f powerwall.yml up -d

You will now need to set up the configs in grafana again: https://github.com/jasonacox/Powerwall-Dashboard#grafana-setup

[mp09]

Cleaning that up worked, thanks!

[mcbirse]

We had some interesting weather today which was great for the beta testing! 😃

Sudden cold front from about midday with clouds & rain (and snow in our alps). Looks to be working well, as below.

However, I noticed I wasn't receiving any rain data (and it was definitely raining).

@jasonacox - I think there is a small issue with retrieving the rain or snow values from the JSON result of the OpenWeather API request. I have made a change to my local test server which seemed to fix the issue, so will submit a PR.

It started raining here at about 12:30pm, so the JSON data began to include "rain.1h" values, however it was showing as Null. I fixed it about 8pm as you can see below.

I've also noticed the JSON response may not include include a 3h value.

I had to change the weather panels to show "1h" instead of "3h" to see rain data - I wonder what would be more reliable as a default?

Also, when either "rain" or "snow" is not being returned at all, the values for rain_1hr rain_3hr etc. will be set to zero be default.

Maybe a default of None would be better? It appears the OpenWeather API could return either 1hr or 3hr data sometimes, so for consistency my thoughts would be to set the values to None when no data is returned. What do you think?

[jasonacox]

Thanks for the fix @mcbirse! I'm open to making more adjustment. The 1hr and 3hr data point did seem like an odd way to provide the data, but we probably need to see more events to determine the best approach?

[jasonacox]

I'm planning on adding a weather section to the bottom of the default dashboard.json. Thoughts?

[jasonacox]

I have merged the weather branch changes into main and ran some tests. If you are the beta (weather) branch, you can move back to main and upgrade:

# switch to main branch
git checkout main

# upgrade to latest
./upgrade.sh

This should not overwrite your weather11.conf file and upgrade.sh will use that but will force an upgrade of weather411. If you have been following this issue but did not install the beta (weather) branch, the upgrade script will prompt you for the OpenWeatherMap information (Key, location and units).

Thanks everyone for all the help. Please post any issue or questions.

[oralallen82]

I have merged the weather branch changes into main and ran some tests. If you are the beta (weather) branch, you can move back to main and upgrade:

# switch to main branch
git checkout main

# upgrade to latest
./upgrade.sh

This should not overwrite your weather11.conf file and upgrade.sh will use that but will force an upgrade of weather411. If you have been following this issue but did not install the beta (weather) branch, the upgrade script will prompt you for the OpenWeatherMap information (Key, location and units).

Thanks everyone for all the help. Please post any issue or questions.

Would this be the procedure for non-beta testers? I'd definitely like to incorporate this data into my model. We've been having wet/overcast days here in North Texas.

[mcbirse]

Would this be the procedure for non-beta testers? I'd definitely like to incorporate this data into my model. We've been having wet/overcast days here in North Texas.

@oralallen82 - if you never changed from the main branch, it should be as simple running upgrade.sh only and following the prompts to install weather411:

# upgrade to latest
./upgrade.sh

I did this on my production set up (as I was testing the weather branch on a separate vm anyway), and the upgrade and install/setup was all very seamless!

[oralallen82]

Thanks I just did the upgrade and setup weather411 going to install the dashboard now.

On Tue, Aug 30, 2022 at 6:34 PM Michael Birse @.***> wrote:

Would this be the procedure for non-beta testers? I'd definitely like to incorporate this data into my model. We've been having wet/overcast days here in North Texas.

@oralallen82 https://github.com/oralallen82 - if you never changed from the main branch, it should be as simple running upgrade.sh only and following the prompts to install weather411:

upgrade to latest

./upgrade.sh

I did this on my production set up (as I was testing the weather branch on a separate vm anyway), and the upgrade and install/setup was all very seamless!

— Reply to this email directly, view it on GitHub https://github.com/jasonacox/Powerwall-Dashboard/issues/42#issuecomment-1232276448, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIDLQFXT4BDYP2BYX3JCNVLV32LABANCNFSM5ZLAI5RA . You are receiving this because you were mentioned.Message ID: @.***>

[oralallen82]

Is there a way to determine if your weather411 is working as expected. I completed setup ~ 30 or so mins ago and I don't see any data.

[jasonacox]

Hi @oralallen82 - Yes, try this:

# see if weather411 is running
docker logs weather411

Check the web based output from weather411: http://localhost:8676/ (where 'localhost' is the box running the docker containers)

[oralallen82]

ERROR: No config file. Fix and Restart

Hi @oralallen82 - Yes, try this:

# see if weather411 is running
docker logs weather411

Check the web based output from weather411: http://localhost:8676/ (where 'localhost' is the box running the docker containers)

ERROR: No config file. Fix and Restart

[jasonacox]

Thanks @oralallen82 - That means the OpenWeatherMap config was never set up. When you ran upgrade.sh did it ask you to set up weather? If so, did you enter the OpenWeatherMap KEY and your location information? If so, that should have created the config file weather/weather411.conf.

You can try it again by running upgrade.sh or by running weather.sh directly.

Alternative, if you see the file weather/weather411.conf, you can try to remove and start weather411:

docker stop weather411
docker rm weather411
docker images | grep weather411 | awk '{print $3}' | xargs docker rmi -f
docker-compose -f powerwall.yml up -d

[oralallen82]

I ran weather.sh manually twice I didn't get it when I did the upgrade.sh I only see the sample file. I'll try the steps above and report back.

[jasonacox]

Interesting. If you didn't get a weather/weather411.conf after you ran weather.sh, that has me concerned. I'm trying to determine if there is any situation where it would fail to create that file. The first thing weather.sh does is copy the weather/weather411.conf.sample file into weather/weather411conf. Did you get any permission or other errors?

Thanks for your help with this. Worse case you can do the manual copy and edit weather411.conf to have your settings, but the script should be doing that for you.

[oralallen82]

I manually updated the .conf file now when I run the docker logs weather411 I get no such container. ./upgrade.sh doesn't do anything. ./weather.sh shows my manually updated file so I know it's visible now.

I may have screwed up the upgrade i ran sudo :( initially.

[jasonacox]

Oh no! I need to add code to prohibit root from running upgrade.sh. I'll add that to my to-do list.

Did you run docker-compose -f powerwall.yml up -d - if so, did it give any errors?

[oralallen82]

Got it figured out. I reran : docker-compose -f powerwall.yml up -d and that reinstalled and I was able to see weather411 logs

[jasonacox]

Awesome! I do think the sudo probably caused the initial issue. I'll fix that.

[oralallen82]

Awesome! I do think the sudo probably caused the initial issue. I'll fix that.

I definitely believe that did it. I didn't get the option to configure weather411 and after running the weather.sh manually it didn't create the weather411.conf

Once you have it setup where users cannot run sudo for upgrade.sh I think no one else would run into that issue.