Multiple cloud information integrations fail after a while

[lancer73]

The problem

Since HA 2024.10 I'm having problems with multiple integrations among which:

• Aurora
• Electricity maps
• NextDNS
• Open-Meteo

After a while (1h+) these integrations become unavailable. Reloading the integration only succeeds after the entities are unavailable for several hours. Else an error is generated. Restarting home assistant immediately after the integrations have failed also works.

I enabled debug logging for these integrations and they all show that a malformed request (invalid POST data most likely) was received by the remote server, causing them to send an unexepected respons after which the entity becomes unavailable.

Log excerpts: NextDNS 2024-10-11 21:20:13.637 ERROR (MainThread) [homeassistant.components.nextdns.coordinator] Error requesting nextdns data: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://api.nextdns.io/profiles/[obfuscated]/analytics/encryption' 2024-10-11 21:20:13.638 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.040 seconds (success: False) 2024-10-11 21:20:13.639 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated]/analytics/ipVersions, method: get, data: None

Aurora 2024-10-11 21:20:15.668 ERROR (MainThread) [homeassistant.components.aurora.coordinator] Error fetching Aurora data: Error updating from NOAA: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://services.swpc.noaa.gov/json/ovation_aurora_latest.json' 2024-10-11 21:20:15.668 DEBUG (MainThread) [homeassistant.components.aurora.coordinator] Finished fetching Aurora data in 0.015 seconds (success: False) 2024-10-11 21:25:13.643 DEBUG (MainThread) [nextdns] Requesting https://e96929.test.nextdns.io, method: get, data: None

The other integrations show similar failure modes. All fail whithin a couple of minutes from eachother.

Fetching the URL mentioned in the error message from the Core docker containter with curl always succeeds, even after the integration failed and when reloading the integration doesn't work.

What version of Home Assistant Core has the issue?

core-2024.10.2

What was the last working version of Home Assistant Core?

core-2024.9.3

What type of installation are you running?

Home Assistant OS

Integration causing the issue

multiple

Link to integration documentation on our website

https://github.com/aio-libs/aiohttp

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2024-10-11 21:15:13.643 DEBUG (MainThread) [nextdns] Requesting https://[obfuscated profile 1].test.nextdns.io, method: get, data: None
2024-10-11 21:15:13.672 DEBUG (MainThread) [nextdns] Response status 200 for https://[obfuscated profile 1].test.nextdns.io
2024-10-11 21:15:13.672 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.030 seconds (success: True)
2024-10-11 21:15:13.876 DEBUG (MainThread) [nextdns] Requesting https://[obfuscated profile2].test.nextdns.io, method: get, data: None
2024-10-11 21:15:13.913 DEBUG (MainThread) [nextdns] Response status 200 for https://[obfuscated profile2].test.nextdns.io
2024-10-11 21:15:13.913 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.037 seconds (success: True)
2024-10-11 21:15:13.983 DEBUG (MainThread) [nextdns] Requesting https://[obfuscated profile3].test.nextdns.io, method: get, data: None
2024-10-11 21:15:14.020 DEBUG (MainThread) [nextdns] Response status 200 for https://[obfuscated profile3].test.nextdns.io
2024-10-11 21:15:14.021 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.038 seconds (success: True)
2024-10-11 21:15:15.877 DEBUG (MainThread) [homeassistant.components.aurora.coordinator] Finished fetching Aurora data in 0.225 seconds (success: True)
2024-10-11 21:20:13.598 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile 1]/analytics/encryption, method: get, data: None
2024-10-11 21:20:13.599 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile 1]/analytics/status, method: get, data: None
2024-10-11 21:20:13.623 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile3]/analytics/encryption, method: get, data: None
2024-10-11 21:20:13.624 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile2]/analytics/ipVersions, method: get, data: None
2024-10-11 21:20:13.636 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile3]/analytics/status, method: get, data: None
2024-10-11 21:20:13.637 ERROR (MainThread) [homeassistant.components.nextdns.coordinator] Error requesting nextdns data: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://api.nextdns.io/profiles/[obfuscated profile 1]/analytics/encryption'
2024-10-11 21:20:13.638 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.040 seconds (success: False)
2024-10-11 21:20:13.639 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile 1]/analytics/ipVersions, method: get, data: None
2024-10-11 21:20:13.642 ERROR (MainThread) [homeassistant.components.nextdns.coordinator] Error requesting nextdns data: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://api.nextdns.io/profiles/[obfuscated profile 1]/analytics/status'
2024-10-11 21:20:13.642 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.043 seconds (success: False)
2024-10-11 21:20:13.646 DEBUG (MainThread) [nextdns] Requesting https://[obfuscated profile 1].test.nextdns.io, method: get, data: None
2024-10-11 21:20:13.655 ERROR (MainThread) [homeassistant.components.nextdns.coordinator] Error requesting nextdns data: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://api.nextdns.io/profiles/[obfuscated profile3]/analytics/encryption'
2024-10-11 21:20:13.655 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.032 seconds (success: False)
2024-10-11 21:20:13.655 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile2]/analytics/status, method: get, data: None
2024-10-11 21:20:13.658 ERROR (MainThread) [homeassistant.components.nextdns.coordinator] Error requesting nextdns data: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://api.nextdns.io/profiles/[obfuscated profile2]/analytics/ipVersions'
2024-10-11 21:20:13.658 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.035 seconds (success: False)
2024-10-11 21:20:13.672 ERROR (MainThread) [homeassistant.components.nextdns.coordinator] Error requesting nextdns data: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://api.nextdns.io/profiles/[obfuscated profile3]/analytics/status'
2024-10-11 21:20:13.672 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.037 seconds (success: False)
2024-10-11 21:20:13.676 ERROR (MainThread) [homeassistant.components.nextdns.coordinator] Error requesting nextdns data: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://[obfuscated profile 1].test.nextdns.io'
2024-10-11 21:20:13.676 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.030 seconds (success: False)
2024-10-11 21:20:13.677 ERROR (MainThread) [homeassistant.components.nextdns.coordinator] Error requesting nextdns data: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://api.nextdns.io/profiles/[obfuscated profile 1]/analytics/ipVersions'
2024-10-11 21:20:13.677 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.038 seconds (success: False)
2024-10-11 21:20:13.680 ERROR (MainThread) [homeassistant.components.nextdns.coordinator] Error requesting nextdns data: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://api.nextdns.io/profiles/[obfuscated profile2]/analytics/status'
2024-10-11 21:20:13.680 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.025 seconds (success: False)
2024-10-11 21:20:13.730 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile2]/analytics/encryption, method: get, data: None
2024-10-11 21:20:13.756 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile3]/analytics/protocols, method: get, data: None
2024-10-11 21:20:13.758 ERROR (MainThread) [homeassistant.components.nextdns.coordinator] Error requesting nextdns data: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://api.nextdns.io/profiles/[obfuscated profile2]/analytics/encryption'
2024-10-11 21:20:13.758 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.029 seconds (success: False)
2024-10-11 21:20:13.779 ERROR (MainThread) [homeassistant.components.nextdns.coordinator] Error requesting nextdns data: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://api.nextdns.io/profiles/[obfuscated profile3]/analytics/protocols'
2024-10-11 21:20:13.779 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.024 seconds (success: False)
2024-10-11 21:20:13.833 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile 1]/analytics/protocols, method: get, data: None
2024-10-11 21:20:13.866 ERROR (MainThread) [homeassistant.components.nextdns.coordinator] Error requesting nextdns data: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://api.nextdns.io/profiles/[obfuscated profile 1]/analytics/protocols'
2024-10-11 21:20:13.867 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.033 seconds (success: False)
2024-10-11 21:20:13.875 DEBUG (MainThread) [nextdns] Requesting https://[obfuscated profile2].test.nextdns.io, method: get, data: None
2024-10-11 21:20:13.886 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile2]/analytics/protocols, method: get, data: None
2024-10-11 21:20:13.909 ERROR (MainThread) [homeassistant.components.nextdns.coordinator] Error requesting nextdns data: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://[obfuscated profile2].test.nextdns.io'
2024-10-11 21:20:13.910 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.034 seconds (success: False)
2024-10-11 21:20:13.913 ERROR (MainThread) [homeassistant.components.nextdns.coordinator] Error requesting nextdns data: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://api.nextdns.io/profiles/[obfuscated profile2]/analytics/protocols'
2024-10-11 21:20:13.914 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.028 seconds (success: False)
2024-10-11 21:20:13.935 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile3]/analytics/ipVersions, method: get, data: None
2024-10-11 21:20:13.967 ERROR (MainThread) [homeassistant.components.nextdns.coordinator] Error requesting nextdns data: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://api.nextdns.io/profiles/[obfuscated profile3]/analytics/ipVersions'
2024-10-11 21:20:13.967 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.032 seconds (success: False)
2024-10-11 21:20:13.981 DEBUG (MainThread) [nextdns] Requesting https://[obfuscated profile3].test.nextdns.io, method: get, data: None
2024-10-11 21:20:14.011 ERROR (MainThread) [homeassistant.components.nextdns.coordinator] Error requesting nextdns data: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://[obfuscated profile3].test.nextdns.io'
2024-10-11 21:20:14.011 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.030 seconds (success: False)
2024-10-11 21:20:15.668 ERROR (MainThread) [homeassistant.components.aurora.coordinator] Error fetching Aurora data: Error updating from NOAA: 400, message="Expected HTTP/:\n\n b''\n ^", url='https://services.swpc.noaa.gov/json/ovation_aurora_latest.json'
2024-10-11 21:20:15.668 DEBUG (MainThread) [homeassistant.components.aurora.coordinator] Finished fetching Aurora data in 0.015 seconds (success: False)
2024-10-11 21:25:13.643 DEBUG (MainThread) [nextdns] Requesting https://[obfuscated profile 1].test.nextdns.io, method: get, data: None
2024-10-11 21:25:13.684 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.042 seconds (success: False)
2024-10-11 21:25:13.876 DEBUG (MainThread) [nextdns] Requesting https://[obfuscated profile2].test.nextdns.io, method: get, data: None
2024-10-11 21:25:13.913 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.037 seconds (success: False)
2024-10-11 21:25:13.981 DEBUG (MainThread) [nextdns] Requesting https://[obfuscated profile3].test.nextdns.io, method: get, data: None
2024-10-11 21:25:14.010 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.029 seconds (success: False)
2024-10-11 21:25:14.673 ERROR (MainThread) [homeassistant.components.co2signal.coordinator] Error fetching co2signal data: Error occurred while communicating to the Electricity Maps API
2024-10-11 21:25:15.878 DEBUG (MainThread) [homeassistant.components.aurora.coordinator] Finished fetching Aurora data in 0.224 seconds (success: False)
2024-10-11 21:30:13.596 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile 1]/analytics/encryption, method: get, data: None
2024-10-11 21:30:13.599 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile 1]/analytics/status, method: get, data: None
2024-10-11 21:30:13.631 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile3]/analytics/encryption, method: get, data: None
2024-10-11 21:30:13.632 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile2]/analytics/ipVersions, method: get, data: None
2024-10-11 21:30:13.633 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.035 seconds (success: False)
2024-10-11 21:30:13.634 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile3]/analytics/status, method: get, data: None
2024-10-11 21:30:13.636 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.040 seconds (success: False)
2024-10-11 21:30:13.638 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile 1]/analytics/ipVersions, method: get, data: None
2024-10-11 21:30:13.642 DEBUG (MainThread) [nextdns] Requesting https://[obfuscated profile 1].test.nextdns.io, method: get, data: None
2024-10-11 21:30:13.664 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile2]/analytics/status, method: get, data: None
2024-10-11 21:30:13.664 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.033 seconds (success: False)
2024-10-11 21:30:13.665 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.033 seconds (success: False)
2024-10-11 21:30:13.666 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.032 seconds (success: False)
2024-10-11 21:30:13.668 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.030 seconds (success: False)
2024-10-11 21:30:13.684 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.042 seconds (success: False)
2024-10-11 21:30:13.688 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.024 seconds (success: False)
2024-10-11 21:30:13.729 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile2]/analytics/encryption, method: get, data: None
2024-10-11 21:30:13.756 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.027 seconds (success: False)
2024-10-11 21:30:13.756 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile3]/analytics/protocols, method: get, data: None
2024-10-11 21:30:13.780 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.024 seconds (success: False)
2024-10-11 21:30:13.835 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile 1]/analytics/protocols, method: get, data: None
2024-10-11 21:30:13.866 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.032 seconds (success: False)
2024-10-11 21:30:13.876 DEBUG (MainThread) [nextdns] Requesting https://[obfuscated profile2].test.nextdns.io, method: get, data: None
2024-10-11 21:30:13.886 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile2]/analytics/protocols, method: get, data: None
2024-10-11 21:30:13.909 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.034 seconds (success: False)
2024-10-11 21:30:13.914 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.028 seconds (success: False)
2024-10-11 21:30:13.936 DEBUG (MainThread) [nextdns] Requesting https://api.nextdns.io/profiles/[obfuscated profile3]/analytics/ipVersions, method: get, data: None
2024-10-11 21:30:13.965 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.029 seconds (success: False)
2024-10-11 21:30:13.982 DEBUG (MainThread) [nextdns] Requesting https://[obfuscated profile3].test.nextdns.io, method: get, data: None
2024-10-11 21:30:14.025 DEBUG (MainThread) [homeassistant.components.nextdns.coordinator] Finished fetching nextdns data in 0.043 seconds (success: False)
2024-10-11 21:30:15.668 DEBUG (MainThread) [homeassistant.components.aurora.coordinator] Finished fetching Aurora data in 0.015 seconds (success: False)

Additional information

Could this be related to recent aiohttp changes?