Scrape: Error while setting up scrape platform for sensor

[syphernl]

The problem

After upgrading to HASS 2023.4.0 (coming from 2023.3.6) the scrape sensor is broken.

What version of Home Assistant Core has the issue?

2023.4.0

What was the last working version of Home Assistant Core?

2023.3.6

What type of installation are you running?

Home Assistant Core

Integration causing the issue

Scrape

Link to integration documentation on our website

https://www.home-assistant.io/integrations/sensor

Diagnostics information

No response

Example YAML snippet

sensor:  
  - platform: scrape
    resource: https://www.tango.nl/stations/tango-XXXXX
    name: "Brandstofprijs Tango XXXXXX"
    select: "#euro95 .pump_price .price"
    unit_of_measurement: €
    scan_interval: 14400

Anything in the logs that might be useful for us?

Traceback (most recent call last):
  File "/srv/hass/lib/python3.10/site-packages/homeassistant/helpers/entity_platform.py", line 304, in _async_setup_platform
    await asyncio.shield(task)
  File "/srv/hass/lib/python3.10/site-packages/homeassistant/components/scrape/sensor.py", line 43, in async_setup_platform
    coordinator: ScrapeCoordinator = discovery_info["coordinator"]
TypeError: 'NoneType' object is not subscriptable

### Additional information

_No response_

[home-assistant[bot]]

Hey there @home-assistant/core, mind taking a look at this issue as it has been labeled with an integration (sensor) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of sensor can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign sensor Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

sensor documentation sensor source _{^{(message by IssueLinks)}}

[amayii0]

Hi, I can confirm the exact same error for a similar use case.

Home Assistant 2023.4.0
Supervisor 2023.04.0
Operating System 9.5
Frontend 20230405.0 - latest

  #####################################################
  # Scrape : Diesel / Super 95
  # Base site : https://carbu.com/belgique/index.php/liste-stations-service/GO/Gosselies/6000/BE_ht_1587
  - platform: scrape
    name: "Cora Chatelineau - Super 95 (E10)"
    resource: https://carbu.com/belgique/index.php/station/cora/chatelineau/6200/127
    select: "h1.price"
    value_template: '{{ value.split(" ")[0] | replace (",", ".") | replace ("-", "0") | float }}'
    unit_of_measurement: "€"
    index: 0
    scan_interval: 21600

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 304, in _async_setup_platform
    await asyncio.shield(task)
  File "/usr/src/homeassistant/homeassistant/components/scrape/sensor.py", line 43, in async_setup_platform
    coordinator: ScrapeCoordinator = discovery_info["coordinator"]
TypeError: 'NoneType' object is not subscriptable

[sohnvonoff]

have a similar error after upgrading to HASS 2023.4.0, but with growatt integration:

2023-04-09 13:50:14.082 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up growatt_server platform for sensor Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 304, in _async_setup_platform await asyncio.shield(task) File "/usr/src/homeassistant/homeassistant/components/growatt_server/sensor.py", line 87, in async_setup_entry devices, plant_id = await hass.async_add_executor_job(get_device_list, api, config) File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/usr/src/homeassistant/homeassistant/components/growatt_server/sensor.py", line 50, in get_device_list user_id = login_response["user"]["id"] KeyError: 'user'

[andriymoroz]

Scrape can now be configured in UI (Device/Integrations) While converting couple of mine noticed small but important issues - it seems '!secrets' cannot be used in UI... and I have to use some passwords in the URL :( UPD: also update_interval. How to set it in UI?..

[amayii0]

Indeed it does work using UI, but I have 18 sensors to re-configure :( 17 left...

Also missing interval I feel like the yaml way is better for many sensors and copy paste, much easier to maintain too :(

After having migrated 4 sensors (2 + 2), do I really need 9 integrations? Also I have to rename integration each time after setting first sensor. It would be great to set its name while defining its URL.

Use case : Rip Diesel (B7) and Super E95 (E10) prices from a website for 9 stations. So 9 times the same URL but 2 values each time. Exact same select and value_template for all, only URL and Index change between station

No way to rename a sensor from UI

[sebadv]

I feel your pain... 39 sensors to do here.

[amayii0]

I feel your pain... 39 sensors to do here.

24 at the end of the day, I took the opportunity to add some more stations and official prices. Not fun, be careful of typos

[justinmaiuto]

I think im having a similar issue since the May 23 update, but doesn't seem related to 'scrape' but rather a custom sensor I think?

Logger: homeassistant.components.sensor
Source: helpers/update_coordinator.py:222
Integration: Sensor (documentation, issues)
First occurred: 04:13:48 (2 occurrences)
Last logged: 23:28:06

Error while setting up sems platform for sensor
Traceback (most recent call last):
  File "/config/custom_components/sems/sensor.py", line 57, in async_update_data
    inverters = result["inverter"]
TypeError: 'NoneType' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 250, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 206, in _async_update_data
    return await self.update_method()
  File "/config/custom_components/sems/sensor.py", line 97, in async_update_data
    raise UpdateFailed(f"Error communicating with API: {err}")
homeassistant.helpers.update_coordinator.UpdateFailed: Error communicating with API: 'NoneType' object is not subscriptable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 304, in _async_setup_platform
    await asyncio.shield(task)
  File "/config/custom_components/sems/sensor.py", line 118, in async_setup_entry
    await coordinator.async_config_entry_first_refresh()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 222, in async_config_entry_first_refresh
    raise ex
homeassistant.exceptions.ConfigEntryNotReady: Error communicating with API: 'NoneType' object is not subscriptable

[Spirituss]

The same issue is here. I have the message on the log:

ERROR (MainThread) [homeassistant.components.sensor] Error while setting up scrape platform for sensor
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 304, in _async_setup_platform
    await asyncio.shield(task)
  File "/usr/src/homeassistant/homeassistant/components/scrape/sensor.py", line 43, in async_setup_platform
    coordinator: ScrapeCoordinator = discovery_info["coordinator"]

And also can't create scrape sensor via UI:

User input malformed: expected a dictionary for dictionary value @ data['headers']

[nathan-gs]

I resolved this by switching from sensor to scrape, see https://www.home-assistant.io/integrations/scrape/

Eg. you need to change

sensor:
  - platform: scrape
    name: test
    resource: https://example.org

to:

scrape:
  - resource: https://example.org
    sensor:
    - name: test
      select: "td"

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.