Opower not working for PSE

[sethRait]

The problem

Continually recieve internal server error when setting up Opower for PSE. Log message is Config entry 'Puget Sound Energy (PSE) (sethrait)' for opower integration not ready yet: 500, message='Internal Server Error', url=URL('https://pse.opower.com/ei/edge/apis/DataBrowser-v1/cws/cost/utilityAccount/<ACCOUNT_NUMBER>?aggregateType=day&startDate=2022-07-10T00:00:00-07:00&endDate=2023-07-08T00:00:00-07:00'); Retrying in background

What version of Home Assistant Core has the issue?

core-2023.8.4

What was the last working version of Home Assistant Core?

NA

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Opower

Link to integration documentation on our website

https://www.home-assistant.io/integrations/opower

Diagnostics information

Logs from running opower's demo.py locally (with certain values redacted):

python : DEBUG:C:\Users\sethr\repos\opower\src\opower\opower.py:Fetching: https://pse.opower.com/ei/edge/apis/multi-account-v1/cws/pse/customers?offset=0&batchSize=100&addressFilter= At line:1 char:1

• python src/demo.py --utility pse --username sethrait --password 8wEiT ...

•   + CategoryInfo          : NotSpecified: (DEBUG:C:\Users\...&addressFilter=:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
  
  

DEBUG:C:\Users\sethr\repos\opower\src\opower\opower.py:Fetched: { "customers": [ { "id": REDACTED_ID, "uuid": "REDACTED_UUID", "legacyOpowerId": "43-1-5ff8aa", "accountNumber": "REDACTED_ACCOUNT_NUMBER", "accountName": "REDACTED_ADDRESS", "address": { "uuid": "", "streetNumber": "", "streetName": "****", "subpremise": null, "postalCode": "*****", "city": "SEATTLE", "country": "US", "state": "WA" }, "type": "RESIDENTIAL", "utilityAccounts": [ { "id": REDACTED_ID, "uuid": "REDACTED_UUID", "utilityAccountId": " REDACTED_ID_2", "utilityAccountId2": null, "servicePointId": REDACTED_ID_3, "meterType": "GAS", "preferredUtilityAccountId": " REDACTED_ID_2", "readResolution": "QUARTER_HOUR" } ] } ], "offset": 0, "batchSize": 100, "total": 1 } DEBUG:C:\Users\sethr\repos\opower\src\opower\opower.py:Fetching: https://pse.opower.com/ei/edge/apis/bill-forecast-cws-v1/cws/pse/customers/REDACTED_UUID/combined-forecast DEBUG:C:\Users\sethr\repos\opower\src\opower\opower.py:Fetched: { "isValidUser": true, "totalForecast": { "meterType": "COMBINED", "startDate": "2023-09-02", "endDate": "2023-10-01", "currentDate": "2023-09-05", "daysInPeriod": 30, "currentDay": 4, "daysLeftInBill": 26, "forecastedUsage": 114, "forecastedCost": 0, "typicalUsage": 169, "typicalCost": 0, "budgetBilling": false, "costToDate": 0, "usageToDate": 8, "currencySymbol": "$" }, "totalMetadata": [ "NO_FORECASTED_COST", "ESTIMATED_PREVIOUS_BILL" ], "accountForecasts": [ { "unitOfMeasure": "THERM", "meterType": "GAS", "startDate": "2023-09-02", "endDate": "2023-10-01", "currentDate": "2023-09-05", "daysInPeriod": 30, "currentDay": 4, "daysLeftInBill": 26, "forecastedUsage": 9, "typicalUsage": 14, "budgetBilling": false, "usageToDate": 1, "currencySymbol": "$", "preferredUtilityAccountId": "REDACTED_ID_2", "accountUuids": [ "REDACTED_UUID" ], "isSolar": false } ] } DEBUG:C:\Users\sethr\repos\opower\src\opower\opower.py:Fetching: https://pse.opower.com/ei/edge/apis/DataBrowser-v1/cws/cost/utilityAccount/REDACTED_UUID?aggregateType=day&startDate=2023-08-29T00%3A00%3A00-07%3A00&endDate=2023-09-06T00%3A00%3A00-07%3A00 DEBUG:C:\Users\sethr\repos\opower\src\opower\opower.py:Fetched: { "servicePointId": "6001816530", "utilityAccountUuid": "REDACTED_UUID", "unit": "THERM", "siteTimeZoneId": "America/Los_Angeles", "reads": [ { "startTime": "2023-08-29T00:00:00.000-07:00", "endTime": "2023-08-30T00:00:00.000-07:00", "value": 0, "readType": "ESTIMATED", "providedCost": 0, "readComponents": [], "isPeakPeriod": false, "rebateAmount": 0, "milesDriven": 0 }, { "startTime": "2023-08-30T00:00:00.000-07:00", "endTime": "2023-08-31T00:00:00.000-07:00", "value": 0.22, "readType": "ESTIMATED", "providedCost": 0, "readComponents": [], "isPeakPeriod": false, "rebateAmount": 0, "milesDriven": 2 }, { "startTime": "2023-08-31T00:00:00.000-07:00", "endTime": "2023-09-01T00:00:00.000-07:00", "value": 0.34, "readType": "ESTIMATED", "providedCost": 0, "readComponents": [], "isPeakPeriod": false, "rebateAmount": 0, "milesDriven": 4 }, { "startTime": "2023-09-01T00:00:00.000-07:00", "endTime": "2023-09-02T00:00:00.000-07:00", "value": 0.48, "readType": "ESTIMATED", "providedCost": 0, "readComponents": [], "isPeakPeriod": false, "rebateAmount": 0, "milesDriven": 6 }, { "startTime": "2023-09-02T00:00:00.000-07:00", "endTime": "2023-09-03T00:00:00.000-07:00", "value": 0.64, "readType": "ESTIMATED", "providedCost": 0, "readComponents": [], "isPeakPeriod": false, "rebateAmount": 0, "milesDriven": 8 }, { "startTime": "2023-09-03T00:00:00.000-07:00", "endTime": "2023-09-04T00:00:00.000-07:00", "value": 0, "readType": "ACTUAL", "providedCost": 0, "readComponents": [], "isPeakPeriod": false, "rebateAmount": 0, "milesDriven": 0 } ], "seriesComponents": [], "ratePlans": [] }

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant[bot]]

Hey there @tronikos, mind taking a look at this issue as it has been labeled with an integration (opower) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of opower can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign opower Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

opower documentation opower source _{^{(message by IssueLinks)}}

[tronikos]

This is likely the same as https://github.com/tronikos/opower/issues/32 and we need to reduce the max_request_days

[christophermichaelshaw]

+1 - Can confirm I am experiencing the same issue as detailed above.

[tronikos]

What's the startDate and endDate in your URL? @sethRait could you run demo.py with debug enabled for start and end date that match the failing request? If it's https://github.com/tronikos/opower/issues/32 could you experiment reducing the max_request_days in the code and if that fixes it, send me a PR?

[333fred]

I'm unable to reproduce the errors OP was getting with my own PSE account, using this (redacted) cli: python3 src/demo.py --utility pse --username <redacted> --password <redacted> --start_date 2023-09-02 --end_date 2023-10-01 --aggregate_type hour

However, I'm also unable to see any info in HA about my PSE account; it was working after initial release, but going in recently and clicking on one of the entities (such as Current bill electric usage to date) popped up a message saying that the integration was no longer providing the sensor. Deleting the integration and readding leaves me in the same state (visually) as OP. Happy to move this to a different issue if you don't think it's related @tronikos, but otherwise I could use some guidance on where to find logs to help figure this out.

[PseudoResonance]

I'm also having this issue. I tried out some different values of max_request_days and it didn't help. I tried a variety of time ranges from a year to a day, with day and hour aggregate types, and the cost was still 0. The only time I saw any cost values was with an aggregate type of bill. Most months were populated with a cost then, however the cost_to_date values for the current bill were still 0 of course.

One thing to note, the PSE website also does not allow you to view daily costs. The daily view is only accessible when on the usage tab, not the costs tab. I believe the only possible solution is probably to get a bill history and calculate the cost per unit manually, then apply that to the daily/hourly data as well as the current bill.

[mattgphoto]

Coming here to also mention this integration isn't working for PSE. I've had it set up for at least a month or so, and there is no information in my Energy dashboard about it. It lists the Source, but 0 kWh for Energy and Grid Total, and $0 cost for both as well.

I just gave reloading the integration a shot, and looks like it's just pulling Current bill electric forecasted usage Current bill electric usage to date

Sensors that all show 0: Current bill electric cost to date Current bill electric forecasted cost Typical monthly electric cost Typical monthly electric usage

I just turned on debug logging and got this bit of info: 2023-12-22 15:30:00.589 DEBUG (MainThread) [homeassistant.components.opower.coordinator] Updating sensor data with: [Forecast(account=Account(customer=Customer(uuid='REDACTED'), uuid='REDACTED', utility_account_id='REDACTED', meter_type=<MeterType.ELEC: 'ELEC'>, read_resolution=None), start_date=datetime.date(2023, 12, 1), end_date=datetime.date(2023, 12, 30), current_date=datetime.date(2023, 12, 22), unit_of_measure=<UnitOfMeasure.KWH: 'KWH'>, usage_to_date=506.0, cost_to_date=0.0, forecasted_usage=751.0, forecasted_cost=0.0, typical_usage=0.0, typical_cost=0.0)] 2023-12-22 15:30:00.589 DEBUG (MainThread) [homeassistant.components.opower.coordinator] Updating Statistics for opower:pse_elec_REDACTED_energy_cost and opower:pse_elec_REDACTED_energy_consumption 2023-12-22 15:30:01.521 DEBUG (MainThread) [homeassistant.components.opower.coordinator] Finished fetching Opower data in 3.814 seconds (success: True)

I can confirm I am able to view energy use by day/hour and cost per month on the PSE website. If I can help somehow (I have no coding knowledge or knowledge of how to PR or anything like that in Github) I am happy to, looks like this hasn't worked since the Opower integration launched.

[LeeThompson]

I'm using the docker version and I can't PSE to work either.

The documentation is rather poor, I'm assuming I use my PSE website credentials with the OPower integration.

In my cause it just keeps failing to initialize without any errors. OPower accepts the credentials and starts trying to initialize, it takes a long time. Even with debug it doesn't really show what's going on.

2024-01-03 14:44:25.777 DEBUG (MainThread) [homeassistant.components.opower.coordinator] Updating Statistics for opower:pse_elec_REDACTED_energy_cost and opower:pse_elec_REDCATED_energy_consumption
2024-01-03 14:44:26.448 DEBUG (MainThread) [homeassistant.components.opower.coordinator] Updating Statistics for opower:pse_gas_REDACTED_energy_cost and opower:pse_gas_REDACTED_energy_consumption
2024-01-03 14:44:26.452 DEBUG (MainThread) [homeassistant.components.opower.coordinator] Updating statistic for the first time
2024-01-03 14:45:19.386 DEBUG (MainThread) [homeassistant.components.opower.coordinator] Finished fetching Opower data in 60.011 seconds (success: False)
2024-01-03 14:45:21.009 DEBUG (MainThread) [homeassistant.components.opower.coordinator] Finished fetching Opower data in 57.756 seconds (success: False)

UPDATE: Deleted and readded, trying to get more information from Home Assistant.

First initialization takes 8 minutes to fail with the Finished fetching Opower data in 60.011 seconds (success: False) message. I'm not sure what I can do to get more debug information.

I'm using my Puget Sound Energy (PSE) website login information. On PSE's site, I can see my usage data.

I changed logger on HA to debug globally and still no useful information, sorry.

[Sean-T-Moore]

Yesterday I added the Opower integration for use with my PSE account and it seems to work okay for me. No errors in the logs and it added my GAS and ELEC accounts. Cost values are all zero, but usage values are correct. I configured my Energy dashboard to include the Opower pse gas and elec consumption entities and I get very nice hourly plots of usage! I checked the HA database and see the hourly data in the statistics table. One thing everyone having trouble should check is that they can see usage data from their PSE web account. It took me several emails to PSE before they fixed something on their end to make that visible. They were sure it was something wrong with my browser at first, but it turned out to be their problem.

[LeeThompson]

One thing everyone having trouble should check is that they can see usage data from their PSE web account. It took me several emails to PSE before they fixed something on their end to make that visible. They were sure it was something wrong with my browser at first, but it turned out to be their problem.

I did, browser dev tools even showed it getting data from pse.opower.com

I wonder why the integration doesn't work for me, though.

[LeeThompson]

I noticed I was running the linuxserver/version of the container so I switched to the official one today. Unfortunately, the same problem persists.

Core version is 2024.1.2

1. I'm able to add the OPower integration service although it seems to take awhile to validate my PSE credentials (~50 seconds). It says it is successful, but then in services it says setup failed.
2. The initialization process takes around 8 minutes to fail, no errors are shown, no exceptions are shown, when it says success: False it claims the process took around a minute.
3. My PSE web login works correctly and shows energy usage etc.

Are there any logging/debug options I can enable to get more information? Is there any way to verify the OPower version?

If I run the stand alone OPower (https://github.com/tronikos/opower) I get usage data:

NOTE: We don't use gas very often anymore so the 0's are normal for us.

DEBUG:C:\Applications\OPower\src\opower\opower.py:Fetching: https://pse.opower.com/ei/edge/apis/multi-account-v1/cws/pse/customers?offset=0&batchSize=100&addressFilter=
DEBUG:C:\Applications\OPower\src\opower\opower.py:Fetching: https://pse.opower.com/ei/edge/apis/bill-forecast-cws-v1/cws/pse/customers/REDACTED/combined-forecast
DEBUG:C:\Applications\OPower\src\opower\opower.py:Ignoring combined-forecast since there is no usage or cost. metadata: ['NO_FORECASTED_COST', 'NO_FORECASTED_USAGE', 'ESTIMATED_PREVIOUS_BILL']

Getting historical data: account= Account(customer=Customer(uuid='REDACTED'), uuid='REDACTED2', utility_account_id='REDACTED_ACCOUNTID', meter_type=<MeterType.ELEC: 'ELEC'>, read_resolution=<ReadResolution.QUARTER_HOUR: 'QUARTER_HOUR'>) aggregate_type= day start_date= 2023-12-31 12:10:14.678458 end_date= 2024-01-07 12:10:14.678458
DEBUG:C:\Applications\OPower\src\opower\opower.py:Fetching: https://pse.opower.com/ei/edge/apis/DataBrowser-v1/cws/utilities/pse/utilityAccounts/REDACTED2/reads?aggregateType=day&startDate=2023-12-31&endDate=2024-01-08
start_time      end_time        consumption     start_minus_prev_end    end_minus_prev_end
2023-12-31 00:00:00-08:00       2024-01-01 00:00:00-08:00       54.489  None    None
2024-01-01 00:00:00-08:00       2024-01-02 00:00:00-08:00       65.166  0:00:00 1 day, 0:00:00
2024-01-02 00:00:00-08:00       2024-01-03 00:00:00-08:00       62.3388 0:00:00 1 day, 0:00:00
2024-01-03 00:00:00-08:00       2024-01-04 00:00:00-08:00       57.8064 0:00:00 1 day, 0:00:00
2024-01-04 00:00:00-08:00       2024-01-05 00:00:00-08:00       57.9972 0:00:00 1 day, 0:00:00
2024-01-05 00:00:00-08:00       2024-01-06 00:00:00-08:00       62.1282 0:00:00 1 day, 0:00:00
2024-01-06 00:00:00-08:00       2024-01-07 00:00:00-08:00       24.2676 0:00:00 1 day, 0:00:00


Getting historical data: account= Account(customer=Customer(uuid='REDACTED'), uuid='REDACTED2', utility_account_id='REDACTED_ACCOUNTID', meter_type=<MeterType.GAS: 'GAS'>, read_resolution=<ReadResolution.QUARTER_HOUR: 'QUARTER_HOUR'>) aggregate_type= day start_date= 2023-12-31 12:10:14.678458 end_date= 2024-01-07 12:10:14.678458
DEBUG:C:\Applications\OPower\src\opower\opower.py:Fetching: https://pse.opower.com/ei/edge/apis/DataBrowser-v1/cws/utilities/pse/utilityAccounts/REDACTED2/reads?aggregateType=day&startDate=2023-12-31&endDate=2024-01-08
start_time      end_time        consumption     start_minus_prev_end    end_minus_prev_end
2023-12-31 00:00:00-08:00       2024-01-01 00:00:00-08:00       0       None    None
2024-01-01 00:00:00-08:00       2024-01-02 00:00:00-08:00       0       0:00:00 1 day, 0:00:00
2024-01-02 00:00:00-08:00       2024-01-03 00:00:00-08:00       0       0:00:00 1 day, 0:00:00
2024-01-03 00:00:00-08:00       2024-01-04 00:00:00-08:00       0       0:00:00 1 day, 0:00:00
2024-01-04 00:00:00-08:00       2024-01-05 00:00:00-08:00       0       0:00:00 1 day, 0:00:00
2024-01-05 00:00:00-08:00       2024-01-06 00:00:00-08:00       0       0:00:00 1 day, 0:00:00

Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x0000029C769711F0>
Traceback (most recent call last):
  File "C:\Python\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "C:\Python\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "C:\Python\lib\asyncio\base_events.py", line 746, in call_soon
    self._check_closed()
  File "C:\Python\lib\asyncio\base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x0000029C769711F0>
Traceback (most recent call last):
  File "C:\Python\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "C:\Python\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "C:\Python\lib\asyncio\base_events.py", line 746, in call_soon
    self._check_closed()
  File "C:\Python\lib\asyncio\base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

If I copy demo.py from the OPower project for the Home Assistant container to use and run OPower directly from a bash terminal window, I get usage data. So I think this is a bug in the HA integration.

(I have no idea what's with the exception at the end, it doesn't occur in the HA container run.)

[tronikos]

Can you try with the latest home assistant release? There was a recent fix that disabled ipv6 for this integration that was causing issues for many

[LeeThompson]

Can you try with the latest home assistant release? There was a recent fix that disabled ipv6 for this integration that was causing issues for many

I can as soon as the official container gets updated.

[tronikos]

Disabling ipv6 was included in 2024.1.1

[LeeThompson]

Disabling ipv6 was included in 2024.1.1

Then that hasn't helped.

Current Version Info Core: 2024.1.2 Frontend: 20240104.0

This seems to be a bug in the integration not in the original OPower project

If I run demo.py from the OPower project on my computer (Windows 10) or in the docker container (using terminal) it works.

If I run the integration, it never successfully initializes and even with debug on I don't get any error messages, just that it failed. (Even with debug on). It's frustrating because if I could at least get it to show me any errors, I might have an idea what it's not happy with.

If someone can even direct how to get this integration to report actual debug information (specifically the error messages), that would help.

[tronikos]

If you search for Opower in http://homeassistant.local:8123/developer-tools/statistics do you see any rows there?

Can you try to replicate https://github.com/home-assistant/core/blob/3a4c64b0a7e4e1763401fabe06fb066f96d150a3/homeassistant/components/opower/coordinator.py#L193-L227 with demo.py using the appropriate flags?

[LeeThompson]

If you search for Opower in http://homeassistant.local:8123/developer-tools/statistics do you see any rows there?

There seem to be some (all 0), which is odd because I've deleted the integration for now.
Is there any way to do a clean reinstall of the OPower integration?

Can you try to replicate (code) with demo.py using the appropriate flags?

PSE doesn't supply costs so probably not.

DEBUG:C:\Applications\OPower\src\opower\opower.py:Ignoring combined-forecast since there is no usage or cost. metadata: ['NO_FORECASTED_COST', 'NO_FORECASTED_USAGE', 'ESTIMATED_PREVIOUS_BILL']

[LeeThompson]

Ok, I re-added OPower and turned on debug on everything to try to isolate what may be making this not work.

I've discovered a couple minor things:

1. It is going back years for usage data, why?
2. It never creates a device or entity. (Not sure if that's normal or not).
3. There is real data in the statistic table for power consumption.
4. As far as HA is concerned OPower never completes setup but OPower continues going back through usage history anyway.

I have gone through the full debug log file and done a grep to build a (hopefully) OPower specific debug log. I have redacted sensitive information. I still have the raw file.

It seems to me that the main issue is that OPower thinks the integration setup is failing (maybe due to lack of cost data). It should probably realize it has usable data.

temp-opower-ha_debug-log_redacted.txt

[tronikos]

To delete previous statistics you can follow steps at https://github.com/home-assistant/core/issues/107182#issuecomment-1879375141

1. That's by design. It's to be able to track how your usage has changed over the years. Per the linked code and documentation:

At the initial setup, the integration pulls historical monthly usage/cost since the account activation. If the utility provides more granular data, it pulls daily usage/cost for the past 3 years and hourly usage/cost for the past 2 months (note: typically, utilities provide only monthly or daily data for gas). After the initial setup, the integration keeps pulling data (twice per day) for the past 30 days to allow for any corrections in the data from the utilities.

2. Yes it's normal. Sensors are created only for utilities that provide forecast data. Per the documentation:

The integration adds the following sensors only if your utility provides forecasted usage/cost

4. If you reload the integration it should resume backfilling data so you could try reloading it until it finishes. Alternatively you locally modify the code to backfill a shorter time period. The reason I told you to run demo with the proper flags to replicate the backfill is to see if there is no time period that's causing issues.

Lacking cost data shouldn't be causing any issues.

[LeeThompson]

To delete previous statistics you can follow steps at #107182 (comment)

1. That's by design. It's to be able to track how your usage has changed over the years. Per the linked code and documentation:

At the initial setup, the integration pulls historical monthly usage/cost since the account activation. If the utility provides more granular data, it pulls daily usage/cost for the past 3 years and hourly usage/cost for the past 2 months (note: typically, utilities provide only monthly or daily data for gas). After the initial setup, the integration keeps pulling data (twice per day) for the past 30 days to allow for any corrections in the data from the utilities.

2. Yes it's normal. Sensors are created only for utilities that provide forecast data. Per the documentation:

The integration adds the following sensors only if your utility provides forecasted usage/cost

4. If you reload the integration it should resume backfilling data so you could try reloading it until it finishes. Alternatively you locally modify the code to backfill a shorter time period. The reason I told you to run demo with the proper flags to replicate the backfill is to see if there is no time period that's causing issues.

Lacking cost data shouldn't be causing any issues.

The odd thing is it's technically working; I can go back through the 'energy' dashboard in HA and it's populated (-2 days and back) but in integrations it still thinks it's failing and initializing. Is it because it doesn't have all the data yet? If that's all it is it should probably not act like it's broken.

In summary it looks like this is working but is saying it's not from the HA's integrations (services) screen.

[tronikos]

Your logs end with:

2024-01-12 13:25:33.443 DEBUG (MainThread) [/usr/local/lib/python3.11/site-packages/opower/opower.py] Fetching: https://pse.opower.com/ei/edge/apis/DataBrowser-v1/cws/utilities/pse/utilityAccounts/REDACTED_PSE_CWS_UTILITY_ACCOUNT_GUID/reads?aggregateType=hour&startDate=2020-06-27&endDate=2020-07-23
2024-01-12 13:25:33.746 DEBUG (MainThread) [/usr/local/lib/python3.11/site-packages/opower/opower.py] Fetching: https://pse.opower.com/ei/edge/apis/DataBrowser-v1/cws/utilities/pse/utilityAccounts/REDACTED_PSE_CWS_UTILITY_ACCOUNT_GUID/reads?aggregateType=hour&startDate=2020-05-31&endDate=2020-06-26
2024-01-12 13:25:33.990 DEBUG (MainThread) [homeassistant.components.opower.coordinator] Finished fetching Opower data in 51.030 seconds (success: False)
2024-01-12 13:25:33.991 DEBUG (MainThread) [homeassistant.config_entries] Config entry 'Puget Sound Energy (PSE) (REDACTED_PSE_USER_ID)' for opower integration not ready yet: 500, message='Internal Server Error', url=URL('https://pse.opower.com/ei/edge/apis/DataBrowser-v1/cws/utilities/pse/utilityAccounts/REDACTED_PSE_CWS_UTILITY_ACCOUNT_GUID/reads?aggregateType=hour&startDate=2020-05-31&endDate=2020-06-26'); Retrying in 80 seconds

so it failed getting hourly data between 2020-05-31 and 2020-06-26 Does that work in demo.py? If you reload the integration does it make progress backfilling? Can you try deleting all opower statistics and start from scratch?

[LeeThompson]

Your logs end with:

2024-01-12 13:25:33.443 DEBUG (MainThread) [/usr/local/lib/python3.11/site-packages/opower/opower.py] Fetching: https://pse.opower.com/ei/edge/apis/DataBrowser-v1/cws/utilities/pse/utilityAccounts/REDACTED_PSE_CWS_UTILITY_ACCOUNT_GUID/reads?aggregateType=hour&startDate=2020-06-27&endDate=2020-07-23
2024-01-12 13:25:33.746 DEBUG (MainThread) [/usr/local/lib/python3.11/site-packages/opower/opower.py] Fetching: https://pse.opower.com/ei/edge/apis/DataBrowser-v1/cws/utilities/pse/utilityAccounts/REDACTED_PSE_CWS_UTILITY_ACCOUNT_GUID/reads?aggregateType=hour&startDate=2020-05-31&endDate=2020-06-26
2024-01-12 13:25:33.990 DEBUG (MainThread) [homeassistant.components.opower.coordinator] Finished fetching Opower data in 51.030 seconds (success: False)
2024-01-12 13:25:33.991 DEBUG (MainThread) [homeassistant.config_entries] Config entry 'Puget Sound Energy (PSE) (REDACTED_PSE_USER_ID)' for opower integration not ready yet: 500, message='Internal Server Error', url=URL('https://pse.opower.com/ei/edge/apis/DataBrowser-v1/cws/utilities/pse/utilityAccounts/REDACTED_PSE_CWS_UTILITY_ACCOUNT_GUID/reads?aggregateType=hour&startDate=2020-05-31&endDate=2020-06-26'); Retrying in 80 seconds

so it failed getting hourly data between 2020-05-31 and 2020-06-26 Does that work in demo.py?

Yes

< If you reload the integration does it make progress backfilling?

Can you try deleting all opower statistics and start from scratch?

I need to learn how to do that first... standby.

[LeeThompson]

If you reload the integration does it make progress backfilling?

It doe snot backfill, because it has never marked as being successfully setup/initialized, it starts over from the beginning and stops when it gets to the May 31st 2020 - June 26th 2020 failure.

Basically when it hits that failure, it stops and marks it all as failed which seems to be a bug or design issue.

Subsequent runs start at the current day again and this just keeps repeating. I may need to disable it as I don't want my utility to block my account.

It should accept usable data, skip failures, backfill if possible but it shouldn't come to a halt if something unexpected happens.

[LeeThompson]

Can you try deleting all opower statistics and start from scratch?

The information I've found on this is:

1. Editing it in the dev tools statistic editor which is a nightmare for a large range.
2. Editing the database directly
3. Nuking everything and starting over

For me the third option is unacceptable, I am only curious about the electricity usage as a 'hey might as well', it is not of much importance to my HA applications.

[LeeThompson]

The integration should probably remember that no cost data is available and stop requesting it in large batches.

[tronikos]

Usage and cost data are requested together.

You can follow https://github.com/home-assistant/core/issues/107182#issuecomment-1879375141 to delete all opower statistics.

Since yours failed at https://pse.opower.com/ei/edge/apis/DataBrowser-v1/cws/utilities/pse/utilityAccounts/REDACTED_PSE_CWS_UTILITY_ACCOUNT_GUID/reads?aggregateType=hour&startDate=2020-05-31&endDate=2020-06-26 Can you run demo.py with: --aggregate_type hour --start_date 2020-05-31 --end_date 2020-06-26?

[tronikos]

Wait I just realized it tried to fetch hourly data for a period way in the past. It should only be fetching hourly data for the past 2 months, daily data for the past 3 years, monthly data for all years. It should be fetching everything in under a couple of minutes.

Opower only keeps granular data for a bit over than 3 years (I think 3.5?) which explains why you got 500 for that time period.

Was the date ever wrong on your system? Delete all opower statistics following https://github.com/home-assistant/core/issues/107182#issuecomment-1879375141 to start from scratch.

[tronikos]

I think your system date was at some point wrong. It fetched and wrote statistics. If there are existing statistics the integration fetches hourly data since the last statistic.

One solution to avoid this is to cap how long back to go at https://github.com/home-assistant/core/blob/3a4c64b0a7e4e1763401fabe06fb066f96d150a3/homeassistant/components/opower/coordinator.py#L246

[LeeThompson]

The system date was never wrong. It's in a docker container and gets time from the host.

[tronikos]

Can you:

1. Remove integration
2. Delete all the opower statistics following https://github.com/home-assistant/core/issues/107182#issuecomment-1879375141
3. Re-add integration