envoy_reader
envoy_reader copied to clipboard
jesserizzo
Enphase integration not working after Envoy updated to D7.0.43 (8ed8ea)
After quickly looking at some threads it appears the user retrieves a token from the Enphase site.
Here is some discussion from the Home Assistant forum.
I don't have access to an Enphase Envoy running D7.x firmware. Even when I access the Enphase site my system does not appear under commissioned envoy. I can only create a token for an uncommissioned envoy. In my test, the JWT payload contains an expiration date of 12 hours after the creation of the token. Not sure about commissioned tokens, as there is some confusion in the forum. A commissioned token had an exp of 7 days, while another was 1 hour.
In any case, even with a token that lasts at most 7 days, the envoy_reader code would need to be enhanced to retrieve a token programmatically. This Issue has some code in Go that could be ported over to Python.
For me, this is really not possible until I get access to an Envoy running D7.x firmware.
There is the token authentication as well as the SSO which works if I login directly to the envoy. The token configuration is a bit of a mystery but it does not appear to be required.
On Mon, Dec 27, 2021, 9:15 PM Greg @.***> wrote:
After quickly looking at some threads it appears the user creates a token from the Enphase http://entrez.enphaseenergy.com/ site.
Here is some discussion https://community.home-assistant.io/t/enhpase-envoy-on-2021-8-with-new-energy-feature/328668?page=18 from the Home Assistant forum.
I don't have access to an Enphase Envoy running D7.x firmware. Even when I access the Enphase http://entrez.enphaseenergy.com/ site my system does not appear under commissioned envoy. I can only create a token for an uncommissioned envoy. In my test, the JWT payload contains an expiration date of 12 hours after the creation of the token. Not sure about commissioned tokens, as there is some confusion in the forum. A commissioned token had an exp of 7 days, while another was 1 hour.
In any case, even with a token that lasts at most 7 days, the envoy_reader code would need to be enhanced to retrieve a token programmatically. This Issue https://github.com/dlmcpaul/EnphaseCollector/issues/22#issuecomment-973656321 has some code in Go that could be ported over to Python.
For me, this is really not possible until I get access to an Envoy running D7.x firmware.
— Reply to this email directly, view it on GitHub https://github.com/jesserizzo/envoy_reader/issues/78#issuecomment-1001833301, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRVSOGJ4WXD5HWT2UK3UTEMTJANCNFSM465KDKNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you commented.Message ID: @.***>
What login do you use for SSO? Is it the Envoy local account like admin or installer? Or is it your login for Enphase Enlighten?
From what I'm reading in all of the forums, is that to access the backend Envoy JSON data a token is needed.
It is the same login for enlighten yes.
On Mon, Dec 27, 2021, 9:30 PM Greg @.***> wrote:
What login do you use for SSO? Is it the Envoy local account like admin or installer? Or is it your login for Enphase Enlighten?
From what I'm reading in all of the forums, is that to access the backend Envoy JSON data a token is needed.
— Reply to this email directly, view it on GitHub https://github.com/jesserizzo/envoy_reader/issues/78#issuecomment-1001836868, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRRCJ2MZIYNW2Z47W43UTEONXANCNFSM465KDKNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you commented.Message ID: @.***>
Unfortunately for me even if I had access to another's Envoy system, my Enlighten credentials would probably not work on their system. If the backend Envoy JSON data is available without a token from the Enphase site and only the Enlighten credentials tied to that system are needed, someone with the system and credentials would have to do some investigation and hack some code together to get the ball rolling.
@enkrypt3d I was looking at the Go code in my previous post and seeing how that can be ported over somehow. But you mentioned that SSO is possible, though all of the screenshots (as I don't have access to an Envoy running D7.x firmware) have the Token text field entry.
I assume this is what happens when going to http://x.x.x.x/production.json?
I was wondering can you post a screenshot showing the SSO? And how it is accessed?
@enkrypt3d I was looking at the Go code in my previous post and seeing how that can be ported over somehow. But you mentioned that SSO is possible, though all of the screenshots (as I don't have access to an Envoy running D7.x firmware) have the Token text field entry.
I assume this is what happens when going to
http://x.x.x.x/production.json?I was wondering can you post a screenshot showing the SSO? And how it is accessed?
Clicking the top button redirects to this page:
https://entrez.enphaseenergy.com/authorize?code_challenge=4bbwml7hgn1MJQWf9nLtyzx892MxjA1RPsPaNimflPw&client_id=envoy-ui-client&redirect_uri=https://192.168.1.69/auth/callback&scope=########&response_type=code&code_challenge_method=S256
The ###### is where the envoy serial # is.
and then this page after auth:
https://192.168.0.5/auth/callback?code=776e446b-c93f-4946-9efd-1ed557a6199f
Let me know if I can provide more details! Thanks
I'm not too sure about the code after I retrieve the token. The below code scrapes a token from the Enphase site, just have to replace the userame aaaaa and password bbbbb with the credentials for the Enphase Enlighten website, and Envoy IP Address x.x.x.x.
After getting the token I'm not sure
import requests
from bs4 import BeautifulSoup
payload = {"username": "aaaaa", "password": "bbbbb"}
session = requests.session()
r = requests.post("https://entrez.enphaseenergy.com/login", data=payload)
print("SiteLogin: " + r.text)
payload = {"uncommissioned": "true", "Site": ""}
cookies = requests.utils.cookiejar_from_dict(
requests.utils.dict_from_cookiejar(r.cookies)
)
print("Cookie: " + str(cookies))
r = requests.post(
"https://entrez.enphaseenergy.com/entrez_tokens", data=payload, cookies=cookies
)
print("TokenSite: " + r.text)
soup = BeautifulSoup(r.text, features="html.parser")
jwt = soup.find("textarea").contents[0]
print("Token: " + jwt)
payload = {"Authorization": "Bearer " + jwt}
r = requests.get("https://x.x.x.x/auth/check_jwt", verify=False, data=payload)
cookies = requests.utils.cookiejar_from_dict(
requests.utils.dict_from_cookiejar(r.cookies)
)
print("AuthSite: " + r.text)
print("AuthCookie: " + str(cookies))
r = requests.get(
"https://x.x.x.x/production.json", verify=False, data=payload, cookies=cookies
)
print("Production: " + r.text)
@enkrypt3d Can you run the code against your Envoy?
It'll probably give an error when retrieving the production page.
How do I do that? I'm not a python expert so I'd need some guidance.
After authenticating, I'm able to hit that production.json:
{"production":[{"type":"inverters","activeCount":34.........
On Sat, Jan 8, 2022 at 9:41 PM Greg @.***> wrote:
@enkrypt3d https://github.com/enkrypt3d Can you run the code against your Envoy?
It'll probably give an error when retrieving the production page.
— Reply to this email directly, view it on GitHub https://github.com/jesserizzo/envoy_reader/issues/78#issuecomment-1008216803, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRULFKTLAM4BEXWCEQ3UVDYUFANCNFSM465KDKNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID: @.***>
Also I think we can ignore the token authentication method. the top option only uses the enlighten manager login....
On Sat, Jan 8, 2022 at 10:13 PM Corey J @.***> wrote:
How do I do that? I'm not a python expert so I'd need some guidance.
After authenticating, I'm able to hit that production.json:
{"production":[{"type":"inverters","activeCount":34,"readingTime":0,"wNow":0,"whLifetime":149221},{"type":"eim","activeCount":1,"measurementType":"production","readingTime":1641697963,"wNow":-0.0,"whLifetime":2547550.432,"varhLeadLifetime":9.708,"varhLagLifetime":1454333.844,"vahLifetime":3614484.427,"rmsCurrent":6.029,"rmsVoltage":248.535,"reactPwr":742.428,"apprntPwr":749.07,"pwrFactor":0.0,"whToday":49074.432,"whLastSevenDays":2498476.432,"vahToday":59799.427,"varhLeadToday":1.708,"varhLagToday":16365.844}],"consumption":[{"type":"eim","activeCount":1,"measurementType":"total-consumption","readingTime":1641697963,"wNow":-57.773,"whLifetime":3317154.764,"varhLeadLifetime":2184853.326,"varhLagLifetime":1462122.356,"vahLifetime":4204961.002,"rmsCurrent":-7.788,"rmsVoltage":248.509,"reactPwr":-1905.093,"apprntPwr":-1935.294,"pwrFactor":-1.0,"whToday":40856.764,"whLastSevenDays":3276307.764,"vahToday":41020.002,"varhLeadToday":23112.326,"varhLagToday":16492.356},{"type":"eim","activeCount":1,"measurementType":"net-consumption","readingTime":1641697963,"wNow":-57.773,"whLifetime":1525239.582,"varhLeadLifetime":2184843.618,"varhLagLifetime":7788.512,"vahLifetime":4204961.002,"rmsCurrent":13.817,"rmsVoltage":248.482,"reactPwr":-1162.665,"apprntPwr":1716.703,"pwrFactor":-0.03,"whToday":0,"whLastSevenDays":0,"vahToday":0,"varhLeadToday":0,"varhLagToday":0}],"storage":[{"type":"acb","activeCount":0,"readingTime":0,"wNow":0,"whNow":0,"state":"idle"}]}
On Sat, Jan 8, 2022 at 9:41 PM Greg @.***> wrote:
@enkrypt3d https://github.com/enkrypt3d Can you run the code against your Envoy?
It'll probably give an error when retrieving the production page.
— Reply to this email directly, view it on GitHub https://github.com/jesserizzo/envoy_reader/issues/78#issuecomment-1008216803, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRULFKTLAM4BEXWCEQ3UVDYUFANCNFSM465KDKNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID: @.***>
Copy the above Python code into a text editor like Notepad and change the username, password, and IP Address placeholders. Save the file as tokentest.py. From the command line, change directory to where you saved the Python file and run python tokentest.py
Assuming you have Python installed
Edit: You might have to install the modules by running from the command line
pip install requests
pip install bs4
@.***:~# python tokentest.py
Traceback (most recent call last):
File "/root/tokentest.py", line 19, in
On Sat, Jan 8, 2022 at 10:19 PM Greg @.***> wrote:
Copy the above Python code into a text editor like Notepad and change the username, password, and IP Address placeholders. Save the file as tokentest.py. From the command line, change directory to where you saved the Python file and run python tokentest.py
Assuming you have Python installed
— Reply to this email directly, view it on GitHub https://github.com/jesserizzo/envoy_reader/issues/78#issuecomment-1008221088, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRQSHKW6TCH3OSEEROTUVD5FJANCNFSM465KDKNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID: @.***>
I wasn't expecting it to fail there. I added some print statements to the output in the above code. Not sure if you want to post the output though. You can email it to me if you feel more comfortable.
Got it - sent you the output in your inbox
On Sat, Jan 8, 2022 at 10:33 PM Greg @.***> wrote:
I wasn't expecting it to fail there. I added some print statements to the output. Not sure if you want to post the output though. You can email it to me if you feel more comfortable.
— Reply to this email directly, view it on GitHub https://github.com/jesserizzo/envoy_reader/issues/78#issuecomment-1008222321, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRWSVY77PUOMTQYVI33UVD6YDANCNFSM465KDKNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID: @.***>
I'm working on the changes but have run into a wall. When using the httpx.AsyncClient() it does not seem to work the same as when using httpx.Client(). When trying to get the token from Enphase I can login, but when I goto the TOKEN_URL page I get an Authentication error. Maybe some session info is not being used/saved?
My current code is located at: https://github.com/gtdiehl/envoy_reader/blob/token_firmware/envoy_reader/envoy_reader.py
Example Usage:
Uncommissioned Envoy
python envoy_reader.py -u [email protected] -p password -c False
Commissioned Envoy
python envoy_reader.py -u [email protected] -p password -c True -i SiteID -s EnvoySerial
So I only need to replace the envoy_reader.py file? Or are there other changes needed?
On Wed, Jan 12, 2022 at 4:53 PM Greg @.***> wrote:
I'm working on the changes but have run into a wall. When using the httpx.AsyncClient() it does not seem to work the same as when using httpx.Client(). When trying to get the token from Enphase I can login, but when I goto the TOKEN_URL page I get an Authentication error. Maybe some session info is not being used/saved?
My current code is located at:
https://github.com/gtdiehl/envoy_reader/blob/token_firmware/envoy_reader/envoy_reader.py
— Reply to this email directly, view it on GitHub https://github.com/jesserizzo/envoy_reader/issues/78#issuecomment-1011485973, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRSGP632IKZ2SLLE5LDUVXZ5XANCNFSM465KDKNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID: @.***>
No don't replace it. It can be run by itself from the command line. But at this point nothing to test as it will fail. I have to investigate how to use the async client and have it behave the same as the sync client that was in the proof of concept code.
python envoy_reader.py
Enter the Envoy IP address or host name, or press enter to use 'envoy' as default: 192.168.196.1.69
Enter the Username for Inverter data authentication, or press enter to use 'envoy' as default: #######@gmail.com
Enter the Password for Inverter data authentication, or press enter to use the default password: ########
Reading...
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:main:HTTP GET Attempt #1: http://192.168.196.1.69/production.json
DEBUG:main:HTTP GET Attempt #2: http://192.168.196.1.69/production.json
DEBUG:main:HTTP GET Attempt #3: http://192.168.196.1.69/production.json
DEBUG:main:HTTP GET Attempt #1: http://192.168.196.1.69/api/v1/production
DEBUG:main:HTTP GET Attempt #2: http://192.168.196.1.69/api/v1/production
DEBUG:main:HTTP GET Attempt #3: http://192.168.196.1.69/api/v1/production
DEBUG:main:HTTP GET Attempt #1: http://192.168.196.1.69/production
DEBUG:main:HTTP GET Attempt #2: http://192.168.196.1.69/production
DEBUG:main:HTTP GET Attempt #3: http://192.168.196.1.69/production
Traceback (most recent call last):
File "/root/envoy_reader.py", line 716, in
It's not looking at HTTPS?
For the new firmware, additional command line arguments have to be used. In my previous comments above two examples are shown for the Uncommisioned and Commissioned Envoy token.
I've released the envoy_reader library with support for firmware 7.x.x.
Also I have made changes to the Home Assistant enphase_envoy sensor integration. I'll leave this Issue open until the HA side has been completed.
Before submitting the PR on the HA side it should probably be tested out. The sensor code is located at: https://github.com/gtdiehl/core/tree/envoy_new_fw/homeassistant/components/enphase_envoy
Copy over the above directory and replace it in your HA server instance.
I don't have an Envoy running this firmware so I would say the more testing the better. The areas I'm concerned with are:
- Both setup methods; Commissioned and Uncommissioned
- Setup using zeroconf (mDNS)
- Adding entities to the HA Energy Dashboard
- Tokens are refreshed after expiration
- Verifying that the Enphase Enlighten website is only requesting tokens only after an actual token expiration
- Commissioned tokens expire every 12 hours
- Uncommissioned tokens expire every 24 hours
Thanks!
@gtdiehl thank you! So glad this is working again! Hopefully the tokens get automatically refreshed. I'll let you know how it looks tomorrow.
It looks like the code when used in Home Assistant, is hitting the Enphase server every polling cycle. Before wide usage this should probably be resolved! I'll give an update once this is resolved.
@gtdiehl I'm running this on my HA instance now, looks like it's working. Hit me up if you want to test anything or check on specific issues :)
@IshmaelObeso Did you setup the Envoy using Commissioned or Uncommisioned?
Also I just updated the HA integration code yesterday so that it should only hit the Enphase server when the token expires. In the earlier code it was hit them every polling cycle!
Commissioned, it works fine so far only thing is it seems to forget the configuration every ~12 hours and I have to reinput the credentials. I'll update the code after the weekend and test it out and let you know if anything pops up. My system also has batteries if you ever want to do testing on those.
Commissioned, it works fine so far only thing is it seems to forget the configuration every ~12 hours and I have to reinput the credentials. I'll update the code after the weekend and test it out and let you know if anything pops up. My system also has batteries if you ever want to do testing on those.
Not having that issue here. Mine also has batteries.