Add websocket API to get list of recorded entities

[bdraco]

related thread https://community.home-assistant.io/t/2023-5-lets-talk/567331/273?u=bdraco

Proposed change

frontend PR https://github.com/home-assistant/frontend/pull/19621

In #90992 we now require a list of entity ids to be passed to history to avoid the cases were systems were getting overloaded because the default was to return everything and that behavior was no longer viable (see the issue queue) now that most systems have hundreds of entities.

Some users have expressed a desire to know which entities are being recorded so they can feed the list back into the history APIs to achieve what they were doing before.

This change adds a websocket API to get list of recorded entities. It may be better to implement this as a REST api since we do not generally document websocket APIs, but I did it as a websocket API since we have generally stopped adding new REST apis except for large binary data.

Type of change

• [ ] Dependency upgrade
• [ ] Bugfix (non-breaking change which fixes an issue)
• [ ] New integration (thank you!)
• [x] New feature (which adds functionality to an existing integration)
• [ ] Deprecation (breaking change to happen in the future)
• [ ] Breaking change (fix/feature causing existing functionality to break)
• [ ] Code quality improvements to existing code or addition of tests

Additional information

• This PR fixes or closes issue: fixes #
• This PR is related to issue:
• Link to documentation pull request:

Checklist

• [x] The code change is tested and works locally.
• [ ] Local tests pass. Your PR cannot be merged unless tests pass
• [ ] There is no commented out code in this PR.
• [ ] I have followed the development checklist
• [ ] I have followed the perfect PR recommendations
• [ ] The code has been formatted using Black (black --fast homeassistant tests)
• [x] Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• [ ] Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• [ ] The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• [ ] New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• [ ] For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
• [ ] Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

• [ ] I have reviewed two other open pull requests in this repository.

[home-assistant[bot]]

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of recorder can trigger bot actions by commenting:

• @home-assistant close Closes the pull request.
• @home-assistant rename Awesome new title Renames the pull request.
• @home-assistant reopen Reopen the pull request.
• @home-assistant unassign recorder Removes the current integration label and assignees on the pull request, add the integration domain after the command.

[alexarch21]

Very interesting ! Going to try this out asap.

[alexarch21]

Finished my testing of this patch on Core 2023.5. This works really well and behaves exactly as expected.

I really hope this will be merged into Core. Not only will it restore functionality lost with home-assistant/core#90992 in a much more flexible way, it also provides a very low latency and resource friendly way to retrieve the recorded entity list. This feature is very useful for frontend cards that provide some kind of UI to let users select entities based on recorder inclusion, like for example alternative history panels. Another use case is the automated creation of history graphs based on user supplied patterns. As an example, it will make alexarch21/history-explorer-card#162 possible in a way that was not possible before (due to the latency of the required database fetch).

[roykrikke]

I hope this patch will be part of Core. As @alexarch21 wrote, this is really helpful!

[bdraco]

This is probably useful for populating the history panel and editor with the list of entities that are actually available in history instead of showing everything.

We should work that into the frontend to see if the api as I threw it together makes sense for that use case before moving this forward

[github-actions[bot]]

There hasn't been any activity on this pull request recently. This pull request has been automatically marked as stale because of that and will be closed if no further activity occurs within 7 days. Thank you for your contributions.

[bdraco]

Since the plan is to use this on the frontend now this should be ready to go.

cc @karwosts

[karwosts]

Can you elaborate a bit how this selects entities? It's not quite working how I expected, though I might be missing something.

When trying to test no matter what I add to recorder excludes, it doesn't seem to change the value returned. If I add an entity to recorder exclude (and restart), it is still returned in this list. If I additionally purge that entity, it is still returned in this list. If I add a domain to excludes, entities from that domain are still returned in this list.

Do I need to flush some metadata, or repack the database, or anything like that to make an entity fully "nonrecorded"?

[bdraco]

You need to remove all history for the entity from the database and restart to make it go away after excluding it

[bdraco]

If the entity still exists in states_meta its considered recorded

sqlite> select * from states_meta;
46|sun.sun
79|zone.home
82|sensor.sun_next_dawn

[bdraco]

You might not need to restart actually. As soon as you purge all states it should delete it from that table well

[karwosts]

Does it need to be a full purge? I'm trying to get rid of this entity, but even after this service call (and restart or no restart), it's still in states_meta.

service: recorder.purge_entities
target:
  entity_id: sensor.date

Config:

recorder:
  exclude:
    domains:
      - automation
      - update
    entity_globs:
      - sensor.sun*
      - weather.*
    entities:
      - sensor.date
      - sensor.last_boot # Comes from 'systemmonitor' sensor platform
      - sun.sun # Don't record sun data

[karwosts]

Looks like only recorder.purge does the necessary cleanup. recorder.purge_entities leaves states_meta intact.

Actually this seems inconsistent sometimes, there are some that don't seem to go away no matter what purges I do.

[bdraco]

Looks like only recorder.purge does the necessary cleanup. recorder.purge_entities leaves states_meta intact.

Actually this seems inconsistent sometimes, there are some that don't seem to go away no matter what purges I do.

purge_entities probably gets it into a bad state. That seems like a bug

[bdraco]

Looks like purge_entity_data needs to call _purge_old_entity_ids before returning True

[bdraco]

https://github.com/home-assistant/core/pull/109344

[karwosts]

Just floating some ideas, but do you think it would be sane/feasible to request the inverse of this list: to get the list of entities excluded from recording?

I'm just guessing that the average user probably does not use excludes, or uses it very sparingly, in which case this API will have to prepare and send over the full list of all entity ids in the system whenever requested.

We had a little discussion in frontend about caching the response, but the caching works much better if we cache the list of excluded entities. I think I could flip it myself after retrieving, but maybe it's easy on your side as well.

[bdraco]

The downside to that approach is that we wouldn't know about entities that we have history for but are no longer in the state machine.

[karwosts]

Just want to make sure I'm clearly understanding what you're saying.

I was initially approaching this as solely how can we remove items from the history picker.

But are you saying we may also have entities which are completely unknown to the frontend in history tables, and we currently don't display them or mention them at all, and if we provided this list of recorded entities, the frontend could learn about them and display them?

[bdraco]

Spot on

For example if you migrate from one power monitoring system to another one you might still have history in the database.

Additionally we might have statistic ids for these entities that are no longer in the state machine but this api won't show those. We might be able to add these to the api later

[karwosts]

Ok. I guess that's a good argument for keeping the interface this way.

[karwosts]

What's somewhat interesting about this is it's finding a lot of junk that I had no idea was still there, it's returning me entities with no history records that I deleted months ago that are mentioned nowhere else.

Not sure if that's addressed fixed by your earlier bugfix, but it's interesting.

Was sort of prototyping a devtools panel that could show this information, not sure if that's too useful or not yet though.

[bdraco]

I think thats super useful. Users ask how to see that data all the time

[karwosts]

Frontend change approved to consume the API, now waiting for backend.