core icon indicating copy to clipboard operation
core copied to clipboard

Log and db flooded by "Task exception was never retrieved"

Open MikOsle opened this issue 2 years ago • 14 comments

The problem

The log and db is being flooded repetitively by the same error "Error doing job: Task exception was never retrieved" every 2-10ms, resulting in rapidly increasing disk and CPU usages. It seems to occur if using a non defined variable of a trigger event.

Below is an example of the number of occurrences in less than one minut.

Logger: homeassistant Source: helpers/template.py:686 First occurred: 09:23:26 (14479 occurrences) Last logged: 09:24:22

The behavior is triggered by many of my automation, which I never have had issues with before. The automation below is just one of them.

What version of Home Assistant Core has the issue?

core-2024.2.1

What was the last working version of Home Assistant Core?

core-2024.1.6

What type of installation are you running?

Home Assistant OS

Integration causing the issue

No response

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

alias: bad_badeværelse
description: ""
trigger:
  - platform: state
    entity_id:
      - sensor.badevaerelse_temp_humidity
    enabled: false
  - type: opened
    platform: device
    device_id: b241aedec2a0ac8e94fa9b5b8f47b7f7
    entity_id: binary_sensor.badevaerelse_vindue_contact
    domain: binary_sensor
    id: vindue
  - platform: event
    event_type: timer.finished
    event_data:
      entity_id: timer.badevaerelse_bad
    id: timer
  - platform: numeric_state
    entity_id: sensor.badevaerelse_bad
    above: 0.005
condition: []
action:
  - choose:
      - conditions:
          - condition: state
            entity_id: input_boolean.badevaerelse_bad
            state: "off"
          - condition: template
            value_template: >-
              {{trigger.from_state.state|float*1.05<trigger.to_state.state|float}}
        sequence:
          - service: input_boolean.turn_on
            data: {}
            target:
              entity_id: input_boolean.badevaerelse_bad
          - service: timer.start
            data:
              duration: "00:30:00"
            target:
              entity_id: timer.badevaerelse_bad
          - service: timer.cancel
            data: {}
            target:
              entity_id: timer.lys_sluk_badevaerelse
      - conditions:
          - condition: or
            conditions:
              - condition: trigger
                id: vindue
              - condition: trigger
                id: timer
        sequence:
          - service: input_boolean.turn_off
            data: {}
            target:
              entity_id: input_boolean.badevaerelse_bad
mode: single

Anything in the logs that might be useful for us?

2024-02-13 07:07:46.009 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'dict object' has no attribute 'from_state' when rendering '{{trigger.from_state.state|float*1.05<trigger.to_state.state|float}}'
2024-02-13 07:07:46.010 WARNING (MainThread) [homeassistant.helpers.script] Error in 'choose[1]' evaluation: In 'template' condition: UndefinedError: 'dict object' has no attribute 'from_state'
2024-02-13 07:07:46.013 INFO (MainThread) [homeassistant.components.automation.bad_badevaerelse] bad_badeværelse: Choose at step 1: choice 2: Running automation actions
2024-02-13 07:07:46.014 INFO (MainThread) [homeassistant.components.automation.bad_badevaerelse] bad_badeværelse: Choose at step 1: choice 2: Executing step call service
2024-02-13 07:07:46.015 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 637, in async_trigger
    and not self._cond_func(variables)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 1009, in if_action
    if check(hass, variables) is False:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 296, in if_or_condition
    if check(hass, variables) is True:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 787, in template_if
    return async_template(hass, value_template, variables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 767, in async_template
    info = value_template.async_render_to_info(variables, parse_result=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 686, in async_render_to_info
    assert self.hass and _render_info.get() is None
AssertionError
2024-02-13 07:07:46.029 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 637, in async_trigger
    and not self._cond_func(variables)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 1009, in if_action
    if check(hass, variables) is False:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 296, in if_or_condition
    if check(hass, variables) is True:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 787, in template_if
    return async_template(hass, value_template, variables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 767, in async_template
    info = value_template.async_render_to_info(variables, parse_result=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 686, in async_render_to_info
    assert self.hass and _render_info.get() is None
AssertionError
2024-02-13 07:07:46.032 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 637, in async_trigger
    and not self._cond_func(variables)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 1009, in if_action
    if check(hass, variables) is False:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 296, in if_or_condition
    if check(hass, variables) is True:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 787, in template_if
    return async_template(hass, value_template, variables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 767, in async_template
    info = value_template.async_render_to_info(variables, parse_result=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 686, in async_render_to_info
    assert self.hass and _render_info.get() is None
AssertionError
2024-02-13 07:07:46.037 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 637, in async_trigger
    and not self._cond_func(variables)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 1009, in if_action
    if check(hass, variables) is False:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 296, in if_or_condition
    if check(hass, variables) is True:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 787, in template_if
    return async_template(hass, value_template, variables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 767, in async_template
    info = value_template.async_render_to_info(variables, parse_result=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 686, in async_render_to_info
    assert self.hass and _render_info.get() is None
AssertionError
2024-02-13 07:07:46.044 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 637, in async_trigger
    and not self._cond_func(variables)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 1009, in if_action
    if check(hass, variables) is False:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 296, in if_or_condition
    if check(hass, variables) is True:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 787, in template_if
    return async_template(hass, value_template, variables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 767, in async_template
    info = value_template.async_render_to_info(variables, parse_result=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 686, in async_render_to_info
    assert self.hass and _render_info.get() is None
AssertionError
2024-02-13 07:07:46.049 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 637, in async_trigger
    and not self._cond_func(variables)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 1009, in if_action
    if check(hass, variables) is False:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 296, in if_or_condition
    if check(hass, variables) is True:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 787, in template_if
    return async_template(hass, value_template, variables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 767, in async_template
    info = value_template.async_render_to_info(variables, parse_result=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 686, in async_render_to_info
    assert self.hass and _render_info.get() is None
AssertionError
.....

Additional information

No response

MikOsle avatar Feb 13 '24 08:02 MikOsle

This has been reported on community forums too:

  • https://community.home-assistant.io/t/assertionerror-in-an-automation-with-state-change-trigger/689140
  • https://community.home-assistant.io/t/getting-the-same-error-every-100ms-on-2024-2-1-and-cannot-locate-cause/688349

The consensus is it looks to be caused by state_changed trigger, in a condition which refers to trigger.

This is a regression introduced in 2024.2.1, and worked in 2024.1.6. It results in wearing off disk (when logged) and high cpu usage.

dotdoom avatar Feb 20 '24 19:02 dotdoom

Any updates on this? I had to rollback to 2024.1.6 from 2024.2.1, and not sure if it’s safe to try updating again atm.

aetha avatar Mar 02 '24 09:03 aetha

I have the same problem. Are there any info, what to do? Pls!

shummo avatar Mar 08 '24 12:03 shummo

I silenced the log message in configuration.yaml using logger section:

logger:
  filters:
    # https://github.com/home-assistant/core/issues/110456
    homeassistant:
      - "Error doing job: Task exception was never retrieved"

Is it great? No. But I don't know how to attract the right attention to this issue, and this saves my ODROID eMMC, which is not replaceable.

dotdoom avatar Mar 09 '24 13:03 dotdoom

Same issue? Thanks for the tip about silencing the logger, wonder if that'll help me...

26tajeen avatar Mar 17 '24 14:03 26tajeen

I think this is caused by the

          - condition: template
            value_template: >-
              {{trigger.from_state.state|float*1.05<trigger.to_state.state|float}}

But now I am wondering, you have 4 triggers (from which 1 is disabled). The template you use seems to expect a certain trigger to have happened, could that be the case?

joostlek avatar Mar 17 '24 16:03 joostlek

I think that template is indeed the cause, as those variables will only exist on certain triggers, but not on the timer trigger. So I expect this will occur when the timer finishes.

However, it should not cause this many logs

TheFes avatar Mar 17 '24 18:03 TheFes

Right so we dove a bit into this and there might be an issue with async code. We can't replicate this ourselves on our system so we are suspecting this is caused by a custom component. Can you reboot in safe mode and check if this still exists?

joostlek avatar Mar 18 '24 07:03 joostlek

Given I’m experiencing the same from a template condition which isn’t properly handling errors itself, would it help to have a list of my custom components?

26tajeen avatar Mar 18 '24 07:03 26tajeen

I am not going to dive into every custom component at this moment. I first want to check if that is actually the reason why this happens.

joostlek avatar Mar 18 '24 08:03 joostlek

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

The AssertionError should normally not be reachable, which indicates you have an integration doing non-threadsafe operations on asyncio objects which is corrupting the internal asyncio state.

When an integration modifies the event loop internals from another thread the behavior is undefined and will usually eventually lead to a crash.

If you put asyncio in debug mode, it will be able to detect some of these bad calls and maybe even block them. You can use the profiler service to do that: https://www.home-assistant.io/integrations/profiler/#service-profilerset_asyncio_debug

If you can't get far enough along you can also enable debugpy: in configuration.yaml instead (but its much slower) https://www.home-assistant.io/integrations/debugpy/

bdraco avatar Apr 21 '24 22:04 bdraco

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.