Accumulated values not reported via mqtt2 anymore

[rene-bayer]

Describe the bug A clear description of what the bug is.

To Reproduce Steps to reproduce the behavior: Use HomeAssistant mqtt2 reporting

Expected behavior Accumulated values should be shown, these are unavailable since 2.1.4

Screenshots

Hardware information:

• Country: Austria
• Meter: Sagemcom
• Encryption enabled yes
• AMS reader: ESP32
• M-bus adapter (if applicable): TSS721 Modul

Relevant firmware information:

• Version: since 2.1.4
• MQTT: yes
• HAN GPIO: GPIO5
• HAN baud and parity: 2400 8E1
• Temperature sensors: NaN
• ENTSO-E API enabled: no

Additional context NaN

[ArnieO]

I believe accumulated data only comes at whole hour - have you waited long enough since you updated? https://github.com/gskjold/AmsToMqttBridge/wiki/Message-formats

[rene-bayer]

I did a clean flash of all 3 versions (2.1.3-2.1.5)

2.1.5 ran a whole day without reporting any data.

2.1.3 reported immediatly

[ArnieO]

On my installation I still receive List 3 as expected shortly after whole hour, running v2.1.5 on a Pow-K+ (ESP32) on a Kamstrup meter, "straight" JSON payload from MQTT.

I assume you use the Home Assistant MQTT payload format. Could you check the content of your MQTT payloads at whole hour using MQTT.fx, MQTT-explorer (http://mqtt-explorer.com/) or similar tool?

Here is List 2, that I receive each 10 seconds:

And here is List 3, received shortly after whole hour, it includes accumulated values:

https://github.com/gskjold/AmsToMqttBridge/wiki/Message-formats

[ArnieO]

@rene-bayer : I changed my MQTT output setting to Home Assistant, and found no accumulated values in any /power payload around whole hour while running v2.1.5.

EDIT: No bug, the data is delivered, see next message.

So there seems to be a bug here related to the Home Assistant integration. Thank you for discovering that!

@gskjold : Where can I find a description of the HA payload formats?

[ArnieO]

The accumulated data arrives in the /energy payload:

So I fail to reproduce your issue.

[gskjold]

This must mean that I have touched something in the parser that makes it loose some data for certain meters. Any chance you can enable debugging and check if there are any errors there?

[philipheyde]

I can confirm I am experiencing the same issue with the Home Assistant MQTT output. I am in Denmark, and am using a Kamstrup meter. I can locate more information when I am home later, if needed.

[nvanggaard]

I think i found something "accidentally" when browsing the logs in homeassistant:

homeassistant | 2022-08-08 17:00:07.511 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.tPI | is_defined }}' homeassistant | 2022-08-08 17:00:07.520 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.tPO | is_defined }}' homeassistant | 2022-08-08 17:00:07.526 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.tQI | is_defined }}' homeassistant | 2022-08-08 17:00:07.531 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.tQO | is_defined }}' homeassistant | 2022-08-08 18:00:07.492 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.tPI | is_defined }}' homeassistant | 2022-08-08 18:00:07.498 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.tPO | is_defined }}' homeassistant | 2022-08-08 18:00:07.501 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.tQI | is_defined }}' homeassistant | 2022-08-08 18:00:07.504 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.tQO | is_defined }}'

From the home assistant documentation, is happens because the json payload doesn't have an attribute called val.

[gskjold]

I think this could be a MQTT buffer size issue that will only trigger under certain conditions. Please try the attached firmware: esp32.zip esp32s2.zip esp8266.zip

[gvenaas]

Tested on an ESP32, but no fix on my device

[gskjold]

Weird... I have not touched the MQTT implementations on these versions, so it must be something with parsing. I am not able to reproduce this problem on my meter, so someone else will have to debug this and find the problem

[gvenaas]

My device failed when going from 2.1.3 to 2.1.5

I am also seeing theese messages in HomeAssistant: Template variable error: 'value_json' is undefined when rendering '{{ value_json.tPI | is_defined }}' for all the accumulating values.

[gvenaas]

Could it be som case sensitivity? I see theese in HA:

2022-08-15 22:24:05.637 INFO (MainThread) [homeassistant.components.mqtt.mixins] Got update for entity with hash: ('sensor', 'ams-esp32_tPI') '{'name': 'Accumulated active import', 'state_topic': 'kamstrup-meter/energy', 'unique_id': 'ams-esp32_tPI', 'object_id': 'ams-esp32_tPI', 'unit_of_measurement': 'kWh', 'value_template': '{{ value_json.tPI | is_defined }}', 'device_class': 'energy', 'device': {'identifiers': ['ams-esp32'], 'name': 'AMS reader', 'model': 'ESP32', 'sw_version': 'v2.1.5', 'manufacturer': 'AmsToMqttBridge', 'configuration_url': 'http://ams-esp32.local/'}, 'state_class': 'total_increasing', 'platform': 'mqtt'}'

2022-08-15 22:24:05.637 INFO (MainThread) [homeassistant.components.mqtt.mixins] Ignoring unchanged update for: sensor.ams_esp32_tpi

Where the unique id is different on "tpi" and "tPI"

[philipheyde]

If it helps, this is the MQTT data received by Home Assistant from AmsToMQTT version 2.1.5: { "lv": "", "id": "", "type": "", "P": 468, "Q": 0, "PO": 0, "QO": 534, "I1": 0.29, "I2": 0.47, "I3": 2.64, "U1": 231, "U2": 230, "U3": 232, "PF": 0.65, "PF1": 0.85, "PF2": 0.88, "PF3": 0.59 }

And the errors messages I see in Home Assistant are: Template variable error: 'value_json' is undefined when rendering '{{ value_json.tPI | is_defined }}' Template variable error: 'value_json' is undefined when rendering '{{ value_json.tPO | is_defined }}' Template variable error: 'value_json' is undefined when rendering '{{ value_json.tQI | is_defined }}' Template variable error: 'value_json' is undefined when rendering '{{ value_json.tQO | is_defined }}'

[gskjold]

HA will receive three different payloads, so this is just one of them. It will not contain the accumulated values. Neither will it be received on the same topic. The one shown in above post is published in {publish topic}/power, while accumulated values are posted in {publish topic}/energy. If you live in Norway (and also some other countries) the last one will only be published on top of every hour.

[philipheyde]

Ok, that makes sense. But the errors actually happen every ~10th second, according to the HA log, so I think it has something to do with this payload. But I'll set up a listener for the /energy payload as well, and see if anything stands out.

Edit: I actually get {topic}/energy data in HA several times a minute. So yes, it's not the /power payload. This is the energy payload I get: {"tPI":60762.74,"tPO":0.00,"tQI":226.31,"tQO":10839.35,"rtc":lu}

[philipheyde]

Could the problem be that the lu-value is not in quotation marks?

Wrong string: {"tPI":60762.74,"tPO":0.00,"tQI":226.31,"tQO":10839.35,"rtc":lu}

Possibly correct string: {"tPI":60762.74,"tPO":0.00,"tQI":226.31,"tQO":10839.35,"rtc":"lu"}

[gskjold]

This should have been a epoch timestamp. The template is incorrect (is %llu and should be %lu), but I am a bit surprised that you didnt get "0lu" or a number in front of lu. In any case, try this version: esp8266.zip esp32s2.zip esp32.zip

[philipheyde]

YES! That seems to work! There are no more errors in the HA log, and the payload can be parsed now:

It'll take a while before this is reflected in HA's energy dashboard, but everything seems to be working as expected with this version.

Thank you so much!

[rene-bayer]

Confirmed - Values are reported to HA correctly again.

Hoping that it's also stable :-)

[gskjold]

Incredible. The file have not been changed since initial implementation... So %llu either worked before, or %l gave a number (with lu behind it) which was correctly parsed, but ignored by HA. Who knows, but good to know that it works :) I will release 2.1.6 soon, I hope it is stable too :)