Prepare for new dsmr_parser

[dupondje]

Breaking change

For users using the 5B version: The previous gas sensor (Gas consumption) will change to Gas consumption mbusX. This als with the change multiple gas sensors are possible. Next to that the power failure count sensors and the voltage sags/swells are removed because they are not reported by the 5B version of the dsmr meter.

Proposed change

A new version of dsmr_parser will get released with some changes for 'Peak Usage'. So this needs to be implemented into HA

https://github.com/ndokter/dsmr_parser/pull/113

Type of change

• [ ] Dependency upgrade
• [ ] Bugfix (non-breaking change which fixes an issue)
• [ ] New integration (thank you!)
• [X] New feature (which adds functionality to an existing integration)
• [ ] Deprecation (breaking change to happen in the future)
• [ ] Breaking change (fix/feature causing existing functionality to break)
• [ ] Code quality improvements to existing code or addition of tests

Additional information

• This PR fixes or closes issue: fixes #
• This PR is related to issue:
• Link to documentation pull request:

Checklist

• [x] The code change is tested and works locally.
• [x] Local tests pass. Your PR cannot be merged unless tests pass
• [X] There is no commented out code in this PR.
• [x] I have followed the development checklist
• [x] The code has been formatted using Black (black --fast homeassistant tests)
• [X] Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• [ ] Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• [ ] The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• [x] New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• [ ] For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
• [ ] Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

• [ ] I have reviewed two other open pull requests in this repository.

Next to that, there is also a new DSMR field that contains 13 values (timestamp and value). It might be cool/nice to have it visual in HA, but the question is how to do that. Cause you will have to create 13 entities? The values are the last 13 peak usage values for the last 13 months.

[home-assistant[bot]]

Hey there @robbie1221, @frenck, mind taking a look at this pull request as it has been labeled with an integration (dsmr) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of dsmr can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Change the title of the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign dsmr Removes the current integration label and assignees on the issue, add the integration domain after the command.

[smignon612]

any forecast on the release date?

[dupondje]

any forecast on the release date?

I hope it will get into 2023.2.0, but depends on if it gets reviewed and merged on time.

[timvdsm]

Let's hope it will, as it's really welcome for Belgian users :-)

[dupondje]

@frenck : CI passes and branch updated to latest dev. Also did some other improvements to the code, see the additional commits.

Its running fine here on my own HA.

I hope this can get merged before the 2023.2.0 release as this is a hot-topic in Belgium and a lot of people will find this very useful!

[gigatexel]

Please make sure CI passes.

Seems fixed.

[timvdsm]

Is there any news :-)? As everything seems to be ready?

[gigatexel]

@frenck could you review this for @dupondje please? Thanks!

[treynaer]

A temporary solution is to download the fork from dupondje. Copy the folder /homeassistant/components/dsmr/ to your custom_components folder. You have to add "version": "1.0.0" to the manifest.json. Reload HA and the component should be live with the changes.

I also changed the name in manifest.json to DSMR Slimme Meter Capaciteitstarief. This will tell me that the component is live.

[gigatexel]

@dupondje I just installed as a custom_component.

Is it intended behavior that this creates a new (second) gas meter? The gas meter before the update is still present but the gas index reading is unavailable. The new one is functional.

[dupondje]

@dupondje I just installed as a custom_component.

Is it intended behavior that this creates a new (second) gas meter? The gas meter before the update is still present but the gas index reading is unavailable. The new one is functional.

This is caused by the fact that the Gas meter now has a valid/correct Serial assigned. And HA creates a new device because of that.

[wlcrs]

In that case, I would suggest to add an async_migrate_entry that updates the unique_id of the existing sensor to the new serial-number based ID with async_update_entry.

[gigatexel]

One more thing. I don't have the sensor for the yearly average.

[gigatexel]

Again, one more thing, @dupondje. Is the way that sensor data is written to the HA-db correct correct for the monthly peak value? I see that the DB is flooded with identical values, with different state_id's. This also causes the state objects not to behave as designed in https://www.home-assistant.io/docs/configuration/state_object/ In my case, last_updated and last_changed, change every few seconds while the sensor state and attributes remain identical.

[treynaer]

@dupondje I get following error withe the latest commit: homeassistant.setup - ERROR - Unable to prepare setup for platform dsmr.sensor: Platform not found (cannot import name 'EntityCategory' from 'homeassistant.const' (/usr/src/homeassistant/homeassistant/const.py)).

[dupondje]

One more thing. I don't have the sensor for the yearly average.

This was not (yet) added because there is no clean way (at least that I know) in HA to add a sensor with a date & value.

Again, one more thing, @dupondje. Is the way that sensor data is written to the HA-db correct correct for the monthly peak value? I see that the DB is flooded with identical values, with different state_id's. This also causes the state objects not to behave as designed in https://www.home-assistant.io/docs/configuration/state_object/ In my case, last_updated and last_changed, change every few seconds while the sensor state and attributes remain identical.

That was at least not changed in this commit. If you think it needs improvements, I think we've better open an issue for this or fix it in another pull req.

@dupondje I get following error withe the latest commit: homeassistant.setup - ERROR - Unable to prepare setup for platform dsmr.sensor: Platform not found (cannot import name 'EntityCategory' from 'homeassistant.const' (/usr/src/homeassistant/homeassistant/const.py)).

Most likely because you should run on the dev branch for that.

[gigatexel]

One more thing. I don't have the sensor for the yearly average.

This was not (yet) added because there is no clean way (at least that I know) in HA to add a sensor with a date & value.

Yearly average is in kW I assume? Not date/time?

[dupondje]

One more thing. I don't have the sensor for the yearly average.

This was not (yet) added because there is no clean way (at least that I know) in HA to add a sensor with a date & value.

Yearly average is in kW I assume? Not date/time?

It's not an yearly average, it's a kind of array with 13 tuples of timestamp (when the peak occured) and the peak value.

[frederikbove]

Can this feature be added in version 2023.3? this is something really important here in belgium

[Dr-know]

Can this feature be added in version 2023.3? this is something really important here in belgium

Still not in 2023.3 release... What is holding up?

[dupondje]

Can this feature be added in version 2023.3? this is something really important here in belgium

Still not in 2023.3 release... What is holding up?

Getting it reviewed, nothing more. Nothing I can do...

[dupondje]

Rebased to current dev

[gigatexel]

Would it be possible to add a config-option to enable/disable the force_update parameter? I have disabled this in my local copy (5B) and this has no negative side effects. On the positive side, this reduces the I/O and the number of database-rows enormously.

[dupondje]

Would it be possible to add a config-option to enable/disable the force_update parameter? I have disabled this in my local copy (5B) and this has no negative side effects. On the positive side, this reduces the I/O and the number of database-rows enormously.

Please move this discussion to some other issue. Thanks!

[wlcrs]

Is it possible to break this PR into smaller parts? I have the impression that this PR is too big to move forward.

My proposal after looking at the current changeset: One PR for the extra average monthly peak value sensor, one which adapts the sensors enabled for the 5B model, one which adds support for the water meter?

I realize that this is a lot of extra effort, but I've had (and still have) PR's sitting around for close to a year because they were to big.

[gigatexel]

Is it possible to break this PR into smaller parts? I have the impression that this PR is too big to move forward.

My proposal after looking at the current changeset: One PR for the extra average monthly peak value sensor, one which adapts the sensors enabled for the 5B model, one which adds support for the water meter?

I realize that this is a lot of extra effort, but I've had (and still have) PR's sitting around for close to a year because they were to big.

I would like to disagree. There is nothing "big" about this PR. It just doesn't get the attention it needs to be merged.

[smignon612]

Is it possible to break this PR into smaller parts? I have the impression that this PR is too big to move forward. My proposal after looking at the current changeset: One PR for the extra average monthly peak value sensor, one which adapts the sensors enabled for the 5B model, one which adds support for the water meter? I realize that this is a lot of extra effort, but I've had (and still have) PR's sitting around for close to a year because they were to big.

I would like to disagree. There is nothing "big" about this PR. It just doesn't get the attention it needs to be merged.

maybe if we asked @frenck one more time nicely or send him sone cupcakes or etc :D

[dupondje]

Added 1 commit to fix duplicate error on restart

[bdraco]

A new version of the upstream lib has been released that fixes some performance issues

At the risk of stating the obvious this PR has a large scope given the major library differences and it's going to be too big for most reviewers to pick up.

In its current form and scope it will need to wait for someone who is a dsmr expert who can test the changes as well as the risk of collateral breakage is higher than most PRs here...which is a very small number of people.

[dupondje]

A new version of the upstream lib has been released that fixes some performance issues

At the risk of stating the obvious this PR has a large scope given the major library differences and it's going to be too big for most reviewers to pick up.

In its current form and scope it will need to wait for someone who is a dsmr expert who can test the changes as well as the risk of collateral breakage is higher than most PRs here...which is a very small number of people.

Thanks for the response. I know there is some newer version of dsmr_parser available, but I wait until the next release is cut to bump it here, as it contains some other useful fixes.

The PR here is in fact rather easy. For electricity it doesn't has big changes. Only for gas/water it has important improvements. I am running the patch for 2 months without any issues.

It might be an idea to make this somehow easily available for testing so others can test? But I think it will only get merged if frenck gets time to review this. It would be a good and highly wanted improvement to the dsmr Integration.