Powerwall-Dashboard icon indicating copy to clipboard operation
Powerwall-Dashboard copied to clipboard

Any way to import all historical data?

Open mikkyo9 opened this issue 2 years ago • 80 comments

It seems the DB is populated from the time of install? Any way to pull in all the history?

mikkyo9 avatar Mar 04 '22 20:03 mikkyo9

Hi @mikkyo9 - good question. The Powerwall doesn't keep that history itself. When you install the "Powerwall-Dashboard" stack, the Telegraf service starts polling the Powerwall to get the metrics. The data is then stored in the DB for Grafana to display. The Tesla App has this historical data because it (Tesla Cloud) started polling the Powerwall when it was commissioned. I'm not aware of a way to pull this historical data into the database and even if we could, it would be missing some data like the String and Temperature data.

jasonacox avatar Mar 05 '22 02:03 jasonacox

You could probably manually download each day's data and then write a script to transfer that information into the influx DB. I wish I could do an aggregate data download with fine granularity. Best I've found is to do a daily download in the phone app, then upload that to my dropbox and then view the info on my desktop.

dmayo305 avatar Mar 05 '22 04:03 dmayo305

@jasonacox - Your abilities greatly outpace mine, so I'm bringing an idea without specific testing. On https://www.teslaapi.io/powerwalls/state-and-settings and https://www.teslaapi.io/energy-sites/state-and-settings there appears to be an API call for historical data using a query period of day, week, month, year (similar to the granularity offered in the app). My thoughts are, could one develop a script with a start date, and then increment by day to download historical data?

dmayo305 avatar Mar 05 '22 15:03 dmayo305

Thanks @dmayo305 - I have been looking at those Tesla cloud APIs (used by the Tesla app) to add features to pyPowerwall (e.g. update battery reserve setting). However, it seems Tesla keeps making breaking changes so I haven't found a good example/library to use. TeslaPy seems to be close but I haven't been able to get it to work yet. If we can figure it out, it wouldn't be too difficult to do what you mention.

jasonacox avatar Mar 07 '22 03:03 jasonacox

I have a sample project that uses TeslaPy to download Powerwall and Solar site data as json files. I imagine it could be an example of how to get the data for adding to the Dashboard, but I don't know if it would need to be reformatted before loading in to the database.

This example uses the "refresh token" method for logging in, so you'll need to generate that first with a 3rd party app or one of the other examples floating around.

fracai avatar Mar 11 '22 16:03 fracai

https://github.com/fracai/tesla-history

The sample project in question

fracai avatar Mar 11 '22 16:03 fracai

Thanks @fracai ! Nice work. Yes I think we could use that to port the data into InfluxDB. What method did you use to get the "refresh token"? If we extend your project to drop that data into the Dashboard DB, we could include instruction on how to make it work.

Ideally, I want to find an automated way to get the token so I can include that in the pyPowerwall library.

jasonacox avatar Mar 12 '22 20:03 jasonacox

I used "Auth for Tesla", an iOS app. There are a few apps listed on this TeslaMate FAQ.

I think an automated method is ideal, but I've gotten the impression that manually retrieving the refresh token is the best method as Tesla frequently requires entering a captcha in order to login.

Fortunately, I think the refresh token is valid for a very long time and, unless you need to fill in data gaps, it shouldn't be necessary to request historical data. It might not be ideal, but I think it'd be acceptable to use a manual process to fill in old data.

Then again, if the refresh token is stored, it'd be possible to make a daily request of historical data to regularly fill in any missing gaps.

fracai avatar Mar 12 '22 23:03 fracai

There's also this python script for retrieving the token, though it does require manual intervention.

https://github.com/bntan/tesla-tokens

fracai avatar Mar 12 '22 23:03 fracai

Any more work being done for historical data? It'd be really cool to pull in all the old data.

jgleigh avatar Apr 25 '22 22:04 jgleigh

I agree. It's not an easy fix, unfortunatelyl. I'm doing more investigation and open to any help. ;)

jasonacox avatar Apr 26 '22 03:04 jasonacox

What I am most interested in is my NEM year data(which started a month ago). So even if I could just import the basic data(to grid, from grid, home usage) as a single point in history it would be useful to fill in the gap from when I started to use powerwall-dashboard.

dvanderb avatar Aug 01 '22 17:08 dvanderb

I have some good news regarding this! 😀

Thanks to @jasonacox for pointing out the TeslaPy module and also @fracai for the sample project using that. I started testing the module and have worked out how to import historical data via the Tesla Owner API (cloud) directly into InfluxDB. 🎉

Having >1yr of historical data that I would certainly love to see in my dashboard was a good motivator. 😊

I'm still working on the program but hopefully will be ready to release in the next week or few. Full time day job doesn't help 😞 - also I've had to brush up on my python knowledge, as these days I'm primarily coding in a language probably most people have never heard of!

So, I have developed this as a command line tool that accepts a start & end datetime range to retrieve history from the Tesla Owner API and import into InfluxDB. @jasonacox I'm thinking submitting a PR to add it to the tools directory seems appropriate?

I'm using the staged authorization process of TeslaPy as it seemed convenient and easy in my opinion. The login to your Tesla account is only required once, as after login a token is cached/saved and it appears to never expire (similar to the Tesla app which stays logged in).

Obviously not all data that the Powerwall-Dashboard logs is available for import. But, the main data shown in the "Energy Usage" and "Power Meters" panels is.

I think I can get the "Monthly Analysis" data working too (still testing). This is stored in the kwh retention policy, but, it is populated from a CQ based on the data in autogen. So I just need to run a query to refresh it after importing history data.

Also, I've been able to extrapolate the grid status from the backup event history, which is nice. 😃

In case there are problems with an import, obviously backups are highly recommended. But, the program will also tag all the data points with a "source=cloud" tag in InfluxDB. This makes it easy to reverse what was imported with a query like DELETE WHERE source='cloud' if something went wrong (and could be further limited to a specific time period if required). This would only delete the imported data, and not affect data that came from your gateway via the dashboard. Also note original dashboard data is never overwritten or changed if you were to enter a datetime range that overlapped with the dashboard data (although I would not recommend - it is best to just fill periods of missing data only).

The history data available from the Tesla cloud is in 5 minute intervals only (15 min for charge percentage), but it seems good enough. Here are some example comparisons.

Below is showing data logged from Powerwall-Dashboard as normal (i.e. every 5 seconds or so): image

Compared with below where data was imported from the Tesla cloud - the graph and values (e.g. Power Meters) are pretty close: image

Example showing imported history data with a backup event - the grid status value is extrapolated from backup event history: image

If you have a data gap / missing data your graph might look like below: image

This can fixed now by importing the history from Tesla cloud for the specific time period: image

So it seems to be working well so far as I can tell, and useful for importing old data from before you installed Powerwall-Dashboard, as well as if your system went down for some reason!

While I'm still working on this, if anyone has any feature requests, questions, tips etc. please send them my way. 😊

(side note: I'm already thinking about fetching historical weather data as well now, which is available from the OpenWeather One Call API 3.0 "timemachine" API call 😉 )

mcbirse avatar Sep 21 '22 03:09 mcbirse

Hi @mcbirse Great work here!!! Please feel free to submit a PR in the /tools directory. I'm sure others would love this too.

jasonacox avatar Sep 21 '22 06:09 jasonacox

I'm looking forward to testing this. I have several small data dropouts as well as several months of data before I got the dashboard setup.

jgleigh avatar Sep 22 '22 20:09 jgleigh

Same here. Take your time to get it where you're happy, but I'm really looking forward to getting a look as well. Thanks for working on it.

fracai avatar Sep 22 '22 22:09 fracai

Update finally - I have been working on the script to import historical data from Tesla cloud over the past few weeks, and believe it is now good to go! :crossed_fingers:

I will submit a PR to add it to the /tools directory shortly.

Hoping for some feedback from the community on this, as so far all testing has been with my system only.

Please see the below instructions for how to use the script (will also add this to the tools README).

The tool can be used to import historical data from Tesla cloud, for instance from before you started using Powerwall-Dashboard. It can also be used to fill in missing data (data gaps) for periods where Powerwall-Dashboard stopped working (e.g. system down, or lost communication with the Local Gateway). By default, the script will search InfluxDB for data gaps and fill gaps only, so you do not need to identify time periods where you have missing data, as this will be done automatically.

NOTE:

  • The available data from Tesla cloud is limited to 5 minute intervals of power usage, and backup events only. It will not be as accurate as data logged by Powerwall-Dashboard, however is still sufficient to provide approximate usage history. The historical data is obtained via the same API as used by the Tesla mobile app, so should be very similar in accuracy to what is displayed in the app.
  • Data imported by this tool will not overwrite existing data, and will update the following panels only: Energy Usage, Grid Status, Power Meters, and Monthly Analysis

To use the script:

  • Install the required python modules: pip install python-dateutil teslapy influxdb
  • Follow the steps below

Setup and logging in to Tesla account

On first use, it is recommended to use the --login option. This will create the config file, save an auth token so you will not need to login again, and then display the energy site details associated with your Tesla account.

# Login to Tesla account
python3 tesla-history.py --login

It will run in an interactive mode. The example below shows the config creation:

Config file 'tesla-history.conf' not found

Do you want to create the config now? [Y/n] Y

Tesla Account Setup
-------------------
Email address: [email protected]
Save auth token to: [tesla-history.auth]

InfluxDB Setup
--------------
Host: [localhost]
Port: [8086]
User (leave blank if not used): [blank]
Pass (leave blank if not used): [blank]
Database: [powerwall]
Timezone (e.g. America/Los_Angeles): Australia/Sydney

Config saved to 'tesla-history.conf'

In most cases, the [default] values will be correct and can be accepted by pressing Enter, however these can be changed if you have a custom setup.

Generally, only your Tesla account email address and your timezone will be required.

After the config is saved, you will be prompted to login to your Tesla account.

This is done by opening the displayed URL in your browser and then logging in:

----------------------------------------
Tesla account: [email protected]
----------------------------------------
Open the below address in your browser to login.

<copy URL to browser> e.g.: https://auth.tesla.com/oauth2/v3/authorize?response_type=code...etc.

After login, paste the URL of the 'Page Not Found' webpage below.

Enter URL after login: <paste URL from browser> e.g.: https://auth.tesla.com/void/callback?code=...etc.

After you have logged in successfully, the browser will show a 'Page Not Found' webpage. Copy the URL of this page and paste it at the prompt.

Once logged in successfully, you will be shown details of the energy site(s) associated with your account:

----------------------------------------
Tesla account: [email protected]
----------------------------------------
      Site ID: 1234567890
    Site name: My Powerwall
     Timezone: Australia/Sydney
    Installed: 2021-04-01 13:09:54+11:00
  System time: 2022-10-13 22:40:59+11:00
----------------------------------------

Once these steps are completed, you should not have to login again.

Basic script usage and examples

To import history data from Tesla cloud for a given start/end period, use the --start and --end options (date/time range is inclusive and in format YYYY-MM-DD hh:mm:ss):

# Get history data for start/end period
python3 tesla-history.py --start "2022-10-01 00:00:00" --end "2022-10-05 23:59:59"

The above example would retrieve history data for the first 5 full days of October.

You can run in test mode first which will not import any data, by using the --test option:

# Run in test mode with --test (will not import data)
python3 tesla-history.py --start "2022-10-01 00:00:00" --end "2022-10-05 23:59:59" --test

Example output:

----------------------------------------
Tesla account: [email protected]
----------------------------------------
      Site ID: 1234567890
    Site name: My Powerwall
     Timezone: Australia/Sydney
    Installed: 2021-04-01 13:09:54+11:00
  System time: 2022-10-13 22:40:59+11:00
----------------------------------------
Running for period: [2022-10-01 00:00:00+10:00] - [2022-10-05 23:59:59+11:00] (4 days, 22:59:59s)

Searching InfluxDB for data gaps (power usage)
* Found data gap: [2022-10-02 11:21:00+11:00] - [2022-10-02 12:41:00+11:00] (1:20:00s)
* Found data gap: [2022-10-04 06:09:00+11:00] - [2022-10-04 06:45:00+11:00] (0:36:00s)
* Found data gap: [2022-10-04 12:29:00+11:00] - [2022-10-04 14:56:00+11:00] (2:27:00s)

Searching InfluxDB for data gaps (grid status)
* Found data gap: [2022-10-02 11:21:00+11:00] - [2022-10-02 12:41:00+11:00] (1:20:00s)
* Found data gap: [2022-10-04 06:09:00+11:00] - [2022-10-04 06:45:00+11:00] (0:36:00s)
* Found data gap: [2022-10-04 12:29:00+11:00] - [2022-10-04 14:56:00+11:00] (2:27:00s)

Retrieving data for gap: [2022-10-02 11:22:00+11:00] - [2022-10-02 12:40:59+11:00] (1:18:59s)
* Loading daily history: [2022-10-02]
Retrieving data for gap: [2022-10-04 06:10:00+11:00] - [2022-10-04 06:44:59+11:00] (0:34:59s)
* Loading daily history: [2022-10-04]
Retrieving data for gap: [2022-10-04 12:30:00+11:00] - [2022-10-04 14:55:59+11:00] (2:25:59s)

Retrieving backup event history
* Creating grid status data: [2022-10-02 11:22:00+11:00] - [2022-10-02 12:40:59+11:00] (1:18:59s)
* Creating grid status data: [2022-10-04 06:10:00+11:00] - [2022-10-04 06:44:59+11:00] (0:34:59s)
* Creating grid status data: [2022-10-04 12:30:00+11:00] - [2022-10-04 14:55:59+11:00] (2:25:59s)

Writing to InfluxDB (*** skipped - test mode enabled ***)
Updating InfluxDB (*** skipped - test mode enabled ***)
Done.

If backup events are identified, this will be shown in the output, and the imported grid status data will include the outages:

----------------------------------------
Tesla account: [email protected]
----------------------------------------
      Site ID: 1234567890
    Site name: My Powerwall
     Timezone: Australia/Sydney
    Installed: 2021-04-01 13:09:54+11:00
  System time: 2022-10-13 22:40:59+11:00
----------------------------------------
Running for period: [2022-04-03 00:00:00+11:00] - [2022-04-20 00:00:00+10:00] (17 days, 1:00:00s)

Searching InfluxDB for data gaps (power usage)
* Found data gap: [2022-04-03 00:00:00+11:00] - [2022-04-20 00:00:00+10:00] (17 days, 1:00:00s)

Searching InfluxDB for data gaps (grid status)
* Found data gap: [2022-04-03 00:00:00+11:00] - [2022-04-20 00:00:00+10:00] (17 days, 1:00:00s)

Retrieving data for gap: [2022-04-03 00:00:00+11:00] - [2022-04-20 00:00:00+10:00] (17 days, 1:00:00s)
* Loading daily history: [2022-04-03]
* Loading daily history: [2022-04-04]
* Loading daily history: [2022-04-05]
* Loading daily history: [2022-04-06]
* Loading daily history: [2022-04-07]
* Loading daily history: [2022-04-08]
* Loading daily history: [2022-04-09]
* Loading daily history: [2022-04-10]
* Loading daily history: [2022-04-11]
* Loading daily history: [2022-04-12]
* Loading daily history: [2022-04-13]
* Loading daily history: [2022-04-14]
* Loading daily history: [2022-04-15]
* Loading daily history: [2022-04-16]
* Loading daily history: [2022-04-17]
* Loading daily history: [2022-04-18]
* Loading daily history: [2022-04-19]
* Loading daily history: [2022-04-20]

Retrieving backup event history
* Creating grid status data: [2022-04-03 00:00:00+11:00] - [2022-04-20 00:00:00+10:00] (17 days, 1:00:00s)
* Found backup event period: [2022-04-19 20:55:53+10:00] - [2022-04-19 22:00:16+10:00] (1:04:23s)
* Found backup event period: [2022-04-19 20:53:39+10:00] - [2022-04-19 20:54:46+10:00] (0:01:07s)
* Found backup event period: [2022-04-08 19:00:14+10:00] - [2022-04-08 19:02:55+10:00] (0:02:41s)
* Found backup event period: [2022-04-08 18:57:32+10:00] - [2022-04-08 18:58:28+10:00] (0:00:56s)
* Found backup event period: [2022-04-08 18:54:56+10:00] - [2022-04-08 18:56:21+10:00] (0:01:25s)
* Found backup event period: [2022-04-04 21:16:45+10:00] - [2022-04-04 21:19:10+10:00] (0:02:25s)

Writing to InfluxDB
Updating InfluxDB
Done.

Since grid status logging was only added to Powerwall-Dashboard in June, you can use the tool to import grid status history from before this time, without affecting your existing power usage data.

The search for missing power usage / grid status data is done independently, so power usage history retrieval will be skipped and the missing grid status data will still be retrieved from Tesla cloud and imported:

----------------------------------------
Tesla account: [email protected]
----------------------------------------
      Site ID: 1234567890
    Site name: My Powerwall
     Timezone: Australia/Sydney
    Installed: 2021-04-01 13:09:54+11:00
  System time: 2022-10-13 22:40:59+11:00
----------------------------------------
Running for period: [2022-04-03 00:00:00+11:00] - [2022-04-14 00:00:00+10:00] (11 days, 1:00:00s)

Searching InfluxDB for data gaps (power usage)
* None found

Searching InfluxDB for data gaps (grid status)
* Found data gap: [2022-04-03 00:00:00+11:00] - [2022-04-14 00:00:00+10:00] (11 days, 1:00:00s)

Retrieving backup event history
* Creating grid status data: [2022-04-03 00:00:00+11:00] - [2022-04-14 00:00:00+10:00] (11 days, 1:00:00s)
* Found backup event period: [2022-04-08 19:00:14+10:00] - [2022-04-08 19:02:55+10:00] (0:02:41s)
* Found backup event period: [2022-04-08 18:57:32+10:00] - [2022-04-08 18:58:28+10:00] (0:00:56s)
* Found backup event period: [2022-04-08 18:54:56+10:00] - [2022-04-08 18:56:21+10:00] (0:01:25s)
* Found backup event period: [2022-04-04 21:16:45+10:00] - [2022-04-04 21:19:10+10:00] (0:02:25s)

Writing to InfluxDB
Done.

Some convenience date options are also available (e.g. these could be used via cron):

# Convenience date options (both options can be used in a single command if desired)
python3 tesla-history.py --today
python3 tesla-history.py --yesterday

Finally, if something goes wrong, the imported data can be removed from InfluxDB with the --remove option. Data logged by Powerwall-Dashboard will not be affected, only imported data will be removed:

# Remove imported data
python3 tesla-history.py --start "YYYY-MM-DD hh:mm:ss" --end "YYYY-MM-DD hh:mm:ss" --remove

For more usage options, run without arguments or with the --help option:

# Show usage options
python3 tesla-history.py --help
usage: tesla-history.py [-h] [-l] [-t] [-d] [--config CONFIG] [--site SITE] [--ignoretz] [--force] [--remove] [--start START] [--end END] [--today] [--yesterday]

Import Powerwall history data from Tesla Owner API (Tesla cloud) into InfluxDB

options:
  -h, --help       show this help message and exit
  -l, --login      login to Tesla account only and save auth token (do not get history)
  -t, --test       enable test mode (do not import into InfluxDB)
  -d, --debug      enable debug output (print raw responses from Tesla cloud)

advanced options:
  --config CONFIG  specify an alternate config file (default: tesla-history.conf)
  --site SITE      site id (required for Tesla accounts with multiple energy sites)
  --ignoretz       ignore timezone difference between Tesla cloud and InfluxDB
  --force          force import for date/time range (skip search for data gaps)
  --remove         remove imported data from InfluxDB for date/time range

date/time range options:
  --start START    start date and time ("YYYY-MM-DD hh:mm:ss")
  --end END        end date and time ("YYYY-MM-DD hh:mm:ss")
  --today          set start/end range to "today"
  --yesterday      set start/end range to "yesterday"

Advanced option notes

  • --debug can be used to enable debug output. This will print the raw responses from Tesla cloud which might be helpful in some circumstances
  • --force option can be used to import data regardless of existing data (i.e. the search for data gaps is skipped). Should not be required normally, but could be useful for testing purposes
  • --remove will remove any previously imported data from InfluxDB for the date/time range, without affecting data logged by Powerwall-Dashboard
  • --ignoretz if your Tesla Powerwall timezone configuration is incorrect, this option may be required. Enabling it will ignore timezone differences between Tesla and InfluxDB. I'm not sure if this would ever happen without more real-world data on installs - could an installer have configured it wrong? The program will give an error if the detected Tesla site timezone doesn't match your configured InfluxDB timezone (please do check you have configured the InfluxDB timezone correctly if you get this error). If the Tesla timezone is in fact wrong (installer error?), I'd be interested to know. Using this option is untested, but I did attempt to account for it in the program by converting datetimes from the configured InfluxDB timezone to the detected Tesla site timezone and vice versa (it should work...)

InfluxDB notes and Analysis Data

I am recording some technical notes below regarding InfluxDB and the Analysis data updates for anyone interested - although mainly because I'm likely to forget why I have done something a certain way in the program if I need to look at this again in the future. :blush:

Importing data:

  • When writing large amounts of data to InfluxDB, I found it is significantly faster to write data points in Line Protocol format, instead of JSON
  • The power usage data retrieved from Tesla cloud is imported into autogen.http however with an added tag of source='cloud'
  • Any existing data points will not be affected at all, as the tags don't match (refer here for an explanation of how InfluxDB handles duplicate data points)
  • This also means, it is easy to remove any imported data by simply specifying WHERE source='cloud' in a DELETE query - only imported data will be removed, as original data will not have this tag
  • When deleting data in InfluxDB, it is important to note it is not possible to remove data from a specific retention policy, instead a DELETE will remove data from all retention policies. However, due to the imported data being tagged with source='cloud' this will not be an issue as the data in the other retention policies will not have this tag

For updating analysis data of InfluxDB (kwh, daily, and monthly retention policies):

  • After data is imported into autogen, the script will currently replicate the SELECT INTO queries used in the CQ's for kwh, daily, and monthly
  • To update the measurement values of existing data points however, it is important to ensure the data points have the exact same timestamp and tags (refer here again), therefore points cannot be tagged with source='cloud' as this would add points (NOTE: some of the CQ's are a sum() of data points, so adding additional points would result in e.g. doubling of values)
  • So as not to affect the analysis data unnecessarily, the tool will only execute the update queries for time ranges where new data points have been imported (+/- the group by time)
  • Finally just noting, there has been some discussion in #87 regarding the accuracy of the analysis data using integral calculations, so the script may need updating once a more accurate method has been determined

Some final notes for anyone using this tool to potentially import a lot of historical data:

If you have a lot of historical data you would like to import, it is probably advisable to split this up into batches of no more than few months at a time. During my testing, when retrieving the daily history data, I found the Tesla cloud site may stop responding if you overload it with too many requests in short periods (this is noted in the TeslaPy module Be aware that Tesla may temporarily block your account if you are hammering the servers too much).

However, this did only occur when trying to request data as quickly as possible. As such, a default 1 second delay between requests has been added to the script (can be changed by adding DELAY = x in the config file), after which I was able to retrieve a full year of historical data in one run without issue.

Please let us know if you find any issues or have questions.

mcbirse avatar Oct 13 '22 23:10 mcbirse

This looks fantastic. I haven't had a chance to test it yet, but I want to soon. I love that you have it designed to search for and fill gaps. That's great. Thanks for your work. I look forward to testing.

fracai avatar Oct 14 '22 01:10 fracai

AMAZING!!! I imported over six months of data and filled a ton of small data gaps. No issues so far. THANKS!!!

jgleigh avatar Oct 14 '22 15:10 jgleigh

Cool! I can't wait to try it out. I assume your work is based on the most current release schema, so I should upgrade first?

mikkyo9 avatar Oct 14 '22 21:10 mikkyo9

Cool! I can't wait to try it out. I assume your work is based on the most current release schema, so I should upgrade first?

@mikkyo9 - You will need to be running at least version 2.3.0 I believe, as that was the release when the CQ for grid status was added (released July). I'd still recommend upgrading to the latest version if you can though, as there has been many fixes.

AMAZING!!! I imported over six months of data and filled a ton of small data gaps. No issues so far. THANKS!!!

@jgleigh -

Awesome to hear, thanks for the feedback!

Yes, you may see a number of small data gaps identified even if you think your system had been logging non-stop without issues.

This is more likely with the grid status data as the gap threshold is 1 minute for that, as apposed to 5 minutes for power usage.

Any time you do an upgrade, for instance, you probably lose a few minutes of data but generally not more than 5 mins in my experience.

I considered accepting longer gaps for grid data - but then looking at my own backup history, I've had a number of events that range from a few seconds to a few minutes... so this way these will still be filled in if the system did not log data during that time for whatever reason.

mcbirse avatar Oct 15 '22 06:10 mcbirse

Great job on this! Worked very well filling dozens of gaps and a missing year of data.

mikkyo9 avatar Oct 15 '22 10:10 mikkyo9

Works like a charm. Thanks for putting this together @mcbirse

youzer-name avatar Oct 15 '22 15:10 youzer-name

Worked quite well for me also, thanks for the all your efforts!

DMBlakeley avatar Oct 15 '22 15:10 DMBlakeley

This worked like a charm. It's amazing what a community can do. I have all the date from system install August 2021 thanks @mcbirse and @jasonacox

oralallen82 avatar Oct 16 '22 22:10 oralallen82

Thanks everyone for the feedback - great to know it is working! 😊

I had one last TODO item for the program, which was to confirm the home power usage calculation is correct during VPP events (it should be 🤞 ).

@cwagz - as discussed earlier, you have some data from a VPP event which should help us to confirm this.

If possible, it would be great if you could run the program with the below arguments when you have a chance:

python3 tesla-history.py --start "2022-08-17 00:00:00" --end "2022-08-17 23:59:59" --force --debug --test

Using the --test option will ensure no data is imported so it won't affect your system.

--debug will show the responses from Tesla cloud. I am interested in seeing the power usage responses only, like below: (i.e. not the other responses, which could include personal information about your powerwall)

        {
            "timestamp": "2022-08-17T18:00:00+10:00",
            "solar_power": 0,
            "battery_power": 567.1428571428571,
            "grid_power": 0,
            "grid_services_power": 0,
            "generator_power": 0
        },
        {
            "timestamp": "2022-08-17T18:05:00+10:00",
            "solar_power": 0,
            "battery_power": 565.7142857142857,
            "grid_power": 0,
            "grid_services_power": 0,
            "generator_power": 0
        },
        {
            "timestamp": "2022-08-17T18:10:00+10:00",
            "solar_power": 0,
            "battery_power": 557.8333333333334,
            "grid_power": 0,
            "grid_services_power": 0,
            "generator_power": 0
        },

Specifically I would like to see the values around 6pm-8pm (and before/after) when the VPP event occurred, so we can see in the response how the grid_services_power value is used and whether that affects how home usage power is calculated:

image

Appreciate the help, thanks.

mcbirse avatar Oct 17 '22 02:10 mcbirse

@mcbirse - Here you go. Thanks for working on this!

    {
        "timestamp": "2022-08-17T17:55:00-07:00",
        "solar_power": 1727.142857142857,
        "battery_power": 0,
        "grid_power": 1146,
        "grid_services_power": 0,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T18:00:00-07:00",
        "solar_power": 1592.857142857143,
        "battery_power": 11457.142857142857,
        "grid_power": -6196.857142857143,
        "grid_services_power": -10547.370702489794,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T18:05:00-07:00",
        "solar_power": 1501.6666666666667,
        "battery_power": 11460,
        "grid_power": -7958.333333333333,
        "grid_services_power": -11602.287772738773,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T18:10:00-07:00",
        "solar_power": 1405.7142857142858,
        "battery_power": 11454.285714285714,
        "grid_power": -10301.714285714286,
        "grid_services_power": -11602.387772738773,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T18:15:00-07:00",
        "solar_power": 1308.5714285714287,
        "battery_power": 11455.714285714286,
        "grid_power": -10424.285714285714,
        "grid_services_power": -11602.337772738774,
        "generator_power": 0
    },
	{
        "timestamp": "2022-08-17T19:30:00-07:00",
        "solar_power": 0,
        "battery_power": 11460.140845070422,
        "grid_power": -3353.760563380282,
        "grid_services_power": -10665.09230304092,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T19:35:00-07:00",
        "solar_power": 0,
        "battery_power": 11461.346153846154,
        "grid_power": -2332.6923076923076,
        "grid_services_power": -11600.851106072108,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T19:40:00-07:00",
        "solar_power": 0,
        "battery_power": 11455.757575757576,
        "grid_power": -7867.212121212121,
        "grid_services_power": -11600.841106072108,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T19:45:00-07:00",
        "solar_power": 0,
        "battery_power": 10071.919191919193,
        "grid_power": -7377.080808080808,
        "grid_services_power": -10440.758995464897,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T19:50:00-07:00",
        "solar_power": 0,
        "battery_power": 0,
        "grid_power": 2383.746835443038,
        "grid_services_power": 0,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T19:55:00-07:00",
        "solar_power": 0,
        "battery_power": 0,
        "grid_power": 6388.857142857143,
        "grid_services_power": 0,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T20:00:00-07:00",
        "solar_power": 0,
        "battery_power": 0,
        "grid_power": 6504.666666666667,
        "grid_services_power": 0,
        "generator_power": 0
    },
    {
        "timestamp": "2022-08-17T20:05:00-07:00",
        "solar_power": 0,
        "battery_power": 0,
        "grid_power": 3018.285714285714,
        "grid_services_power": 0,
        "generator_power": 0

cwagz avatar Oct 20 '22 00:10 cwagz

@mcbirse - I just loaded all my history. This is amazing! Thank you again for doing this.

cwagz avatar Oct 20 '22 01:10 cwagz

Does anyone else have some crazy data in the Tesla cloud? This is mine:

image

The sun must have really been blasting!

Would this be easy to clean up manually?

cwagz avatar Oct 20 '22 01:10 cwagz

The sun must have really been blasting!

It's hard to read your tone here - I take it that you are having a chuckle about unexpected and unlikely milliwatt hours values in the data (and that was probably not a heavy cloud/rain day with minimal generation, also not Megawatt hours?)

You can use TeslaPy to take a look historical data - used by mcbirse in his history tool, and I'm having a putter with it on the historical cumulatives in the cloud.

BuongiornoTexas avatar Oct 20 '22 01:10 BuongiornoTexas