ebusd-configuration Improve i18n/COMMENT handling in message definition files

I am struggling with how message definitions are being described for human beings. The information contained in a message being transmitted is outlined by the NAME and various COMMENT columns.

Sometimes this is helpful but it also gives one a hard time trying to figure out what we get from a message named "IFCFault" ;)

    IFCFault                                :    8 occurences in ebusd-2.1.x/
     comments: ['IFC error', 'SD_IFC_Fault_DK']

NAME and COMMENT values in languages other than german contain obvious translation errors e.g. "aktuell" (current) VS "actual" (tatsächlich):

    Time                                    :   38 occurences in ebusd-2.1.x/
     comments: ['Actual Time', 'Aktuelle Uhrzeit', 'Aktuelle Zeit', 'System Time', 
     'Time', 'Uhrzeit', 'Zeit', 'current time', 'time']

    ActualRoomTempDesiredHc1                :   14 occurences in ebusd-2.1.x/
     comments: ['ActualRoomTempSetpointHC1', 'Raumsolltemp. HK1', 
     'current room setpoint considering all basic conditions passed to the control algorithms']

For a lot of NAMEs(or templates) there are multiple COMMENT values with minimal differences:

    StorageTemp                             :   28 occurences in ebusd-2.1.x/
     comments: ['Aktuelle Speichertemperatur und Sensorstatus',
     'Aktuelle Temperatur des Warmstartspeichers (bei Kombigeräten) oder aktuelle Speichertemperatur bei (VC Geräten)',
     'Current temperature for warm start sensor (combi boiler) / Current storage tank sensor (system boiler only)',
     'current storage tank sensor temperature',
     'current temperature for warm start sensor (combination boiler only) Current storage tank sensor (system boiler only)',
     'current temperature for warm start sensor (combinations boiler only) Current storage tank sensor (system boiler only)',
     'd.04 Speicheristtemp',
     'd.04 Speicheristtemp.',
     'd.04 Storage temperature',
     'd.04 Temp storage / she']

    HcName7                                 :    4 occurences in ebusd-2.1.x/
     comments: ['Name HK7', 'Name für HK7', 'Name für den 8. Heizkreis', 'aaucHcNames CH7', 
     'aaucHcNames_7', 'name of CH7', 'name of heating circuit No 7']

The approach of having message definition files per language is error prone and a maintenance nightmare. Message definition changes will result in the need of updating n files and forgetting to update one of them is easy and may lead to hard to spot errors. The suggested simplification of adding a new column for each language (e.g. COMMENT.en) will require the addition of a column to each row in the message definition file. This will make the CSV files even harder to read and maintain. Also there are a lot of german portions in NAME and COMMENT columns of message definition files residing in en/ subfolder and vice versa.

In order to mitigate the aformentioned issues I would suggest to separate message definitions from COMMENT/translations by creating a "single source of truth" for this kind of values. This could be done by using a i18n library like gettext or having a file providing key/value pairs using e.g. the NAME column contents as key. (As the NAME column values may not be unique the key could also be prefixed with CIRCUIT column value)

Ideally this would remove the necessity of having language specific message definition files altogether. If this is not feasible at all the compilation of language specific message definition files could be accomplished by either a build step or on the fly when delivering configuration files using https://cfg.ebusd.eu.

Would you be interested in implementing the described changes? I can help developing the concept and with data extraction but I am refusing to touch C++ ;)

Jan 06 '22 18:01 doubleO8

ebusd already supports the use of multi-lang columns using a suffix (e.g. ".en"), but I just did't have the time to transform all messages here. furthermore, a little tool would be needed doing that as I assume there are many users using their own definitions. of course some language spec file would be nice, but I just lack the time. a disadvantage of that is the more complex use for people adding their own definitions.

Jan 09 '22 09:01 john30

Hi John,

I'd love to help here. I too see the problem of "esoteric" (in the sense of not self-explanatory) configuration csv with its many columns, layers and languages. I started the homeAssistant ebusd add-on which I'm really glad has found an avid community of contributors. My ebusd momentum stalled however when getting to the CSV configs. I have new(er) hardware/software in my homes ebus installation that arn't covered by the current config CSV files and I would love to extend the previous development momentum into expanding the config CSVs.

I've also thought of extending the Tooling for the config files.

i have two requests in this direction: 1 - Would it be possible to get a few more examples or more extended documentation on how the config files work? 2 - Would it be possible to send unknown commands to the JSON output (mqtt messages) via a command line option, possibly with a direct decode output as part of the JSON

May 09 '22 13:05 LukasGrebe

I've started with a grammar definition, generator and syntax aid for vscode in order to get rid of the CSV finally. at the same time I'm working on a UI and normalizing the CSVs. Once the grammar is ready, the normalization is supposed to convert the CSV to the grammar.

besides, the wiki has a lot of info about messages definition, defaults, type templates and more. there is also a kind of walkthrough for creating new CSVs in the howtos.

May 09 '22 19:05 john30

That sounds great! I was also looking at VS code plugins/extensions as a scaffold for tooling 😊

Ive gone through the materials - i think my biggest hurdle was what felt like a large mental load of putting everything together.

An interactive UI showing a few things would make it easier. I was thinking maybe any of the following:

recieved, unknown messages
looked-up hints or substitutions for known elements such as make/model of message master or slave
"syntax" highlighting for parts of a message
maybe previous messages by the same sender/recipient
all messages with the same Primary and or secondary command byte (to compare possible debug values against)
possible debug values
commenting functions
community help eg via semi automatic issues in thsi repository tied into a VS code plugin - eg someone else has the same make/model/software and already opend an issue or pullrequest for a specific command: the vs code plugin could create a link to it using a defined issue syntax. Maybe even pull issue comments or pull-request defenitions right into VS Code via the guthub API

May 09 '22 21:05 LukasGrebe

Hey @doubleO8 - might i asked why this issue is now closed? I'm not seeing any obvious changes as discussed here (Grammar defenition, generator, syntax aid, (script for moving to) language speciffic comment columns.

@john30 should we open a ticket for each of these? Any chance of looking into your work on these elements? I've created a sqlite db to collect unknown messages over a long time frame and hopefully apply any of the other tools to the data inside to quickly come up with suggestions on possible meanings of these unknown messages.

my device a Vaillant 76VWZIO is not in the configuration files and i'd love to expand the config files.

Nov 16 '22 14:11 LukasGrebe