openhab-core
openhab-core copied to clipboard
Yaml config enhancements to support variable substitutions, includes and packages
Changes in this PR:
- Exclude *.inc.yaml from being processed by the YamlModelRepositoryImpl. These are meant to be used as files
!included as packages. - Add a preprocessor for the YAML config file to enhance its syntax to support:
- Variable definition and substitutions / interpolation
- File inclusion with variables that can be interpolated within the included file
- Package files that define a combination of any current and future supported elements (e.g. things, items, custom tags) that make up a device.
- YAML Anchor & insertion
The output of the preprocessor is the same YAML structure and elements that are currently supported without the addition of this PR.
Current flow:
- YAML File -> Yaml Model Repository loader (YamlModelRepositoryImpl)
New flow:
- YAML File -> YamlPreprocessor -> Yaml Model Repository loader (YamlModelRepositoryImpl)
Additional "pseudo elements" which, after being processed, are removed in the generated data output:
variablespackages
Example:
main.yaml:
version: 1 # this can be whatever version currently supported, irrelevant to this PR
# Enhancement sections: variables and packages
variables:
default_bridge: mqtt:broker:mosquitto
housename: The House of Quark
name: default # This can be overridden with !include { vars: { name: ... } }
loungeroom: &LOUNGEROOM Lounge Room # This is a YAML Anchor - it only works in this file
# whereas vars are passed to the included file
# variables defined here will override variables defined in the included files
VARNAME: &VARNAME vars
# the next two variables are to demonstrate nested var interpolation
lounge: lounge
room: room
thingid: default-thing-id # this doesn't make sense but is just an example
packages:
livingroom-light1: !include
file: esphome-light.inc.yaml
vars:
# vars defined here override the main vars defined in the `variables` section above
# but only apply to this included file
thingid: livingroom-light1
name: Living_Room_Light_1
label: Living Room Light 1
location: LivingRoom # This is the name of semantic location group
livingroom-light2: !include
# include the same file, but with different vars
file: esphome-light.inc.yaml
vars:
thingid: livingroom-light2
name: Living_Room_Light_2
label: Living Room Light 2
location: LivingRoom
# The keys here aren't used, as long as they are unique,
# but we can use anchors and aliases to avoid repetition if you like
&BEDROOM_LIGHT bedroom-light: !include
file: esphome-light.inc.yaml
vars:
thingid: *BEDROOM_LIGHT
name: Bedroom_Light
label: Bedroom Light
location: BedRoom
# Our standard YAML File structure
# it can define new things, or when the name matches the one in a package,
# it will combine and override matching properties defined in the package
things:
mqtt:topic:one:
label: a ${housename} - One
bridge: ${default_bridge}
location: Living Room
mqtt:topic:two:
label: b ${housename} - Two
bridge: ${default_bridge}
location: 'singlequoted strings are ${notinterpolated}'
mqtt:topic:three: !include mqtt_template.inc.yaml # simple include - inherit main variables
# full include with vars - compact syntax
mqtt:topic:four: !include { file: mqtt_template.inc.yaml, vars: { name: four, prefix: d, thingid: thing-four } }
# full include, expanded syntax
mqtt:topic:five: !include
file: mqtt_template.inc.yaml
*VARNAME: # Aliases even work as keys although vscode highlighting gets confused
name: *LOUNGEROOM #anchors and aliases work everywhere in the same file
prefix: e
thingid: thing-five
mqtt:topic:six: !include
file: mqtt_template.inc.yaml
vars:
name: six
prefix: f
# interpolations can be nested up to 10 levels deep (just an arbitrary limit)
location: ${${lounge}${room}} # => ${loungeroom} => Lounge Room
thingid: thing-six
# This matches the package, which allows you to override the package
mqtt:topic:livingroom-light1:
config:
payloadAvailable: active
payloadNotAvailable: inactive
mqtt_template.inc.yaml
in this file it starts from the top level (zero indentation), but it gets inserted into whatever level !include was invoked, so it can be as simple as !includeing a plain string, or a full on yaml structure.
label: ${prefix} Included template for ${name}
bridge: ${default_bridge}
location: ${location}
config:
availabilityTopic: tuya/${thingid}/status
payloadAvailable: online
payloadNotAvailable: offline
channels:
power:
type: switch
config:
stateTopic: tuya/${thingid}/state
commandTopic: tuya/${thingid}/command
dimmer:
type: dimmer
config:
stateTopic: tuya/${thingid}/white_brightness_state
commandTopic: tuya/${thingid}/white_brightness_command
esphome-light.inc.yaml
things:
mqtt:topic:${thingid}:
bridge: ${default_bridge}
label: ${label}
config:
availabilityTopic: ${thingid}/status
payloadAvailable: online
payloadNotAvailable: offline
channels:
power:
type: switch
config:
stateTopic: ${thingid}/light/state
commandTopic: ${thingid}/light/command
transformationPattern:
- JSONPATH:$.state
formatBeforePublish: '{"state":"%s"}'
dimmer:
type: dimmer
config:
stateTopic: ${thingid}/light/state
commandTopic: ${thingid}/light/command
transformationPattern:
- JSONPATH:$.brightness
formatBeforePublish: '{"state":"ON","brightness":"%s"}'
min: 0
max: 255
step: 1
# Create three items for this package
items:
${name}: # The main equipment
type: Group
label: ${label}
groups:
- ${location}
tags:
- Lightbulb
${name}_Power:
type: Switch
label: ${label} Power
icon: light
autoupdate: false
groups:
- ${name}
tags: [Switch, Power]
metadata:
alexa:
value: PowerState
ga:
value: lightPower
channels:
# This part isn't yet done I think?
${name}_Dimmer:
type: Dimmer
label: ${label} Dimmer
icon: light
autoupdate: false
groups:
- ${name}
tags: [Control, Brightness]
metadata:
alexa:
value: Brightness
ga:
value: brightness
channels:
# This part isn't yet done I think?
Variables
Supported syntax (subject to change):
${var}returns the var, or a blank string if var is not defined${var-default_value}- returnsdefault_valueonly if var is undefined${var:-default_value}- returnsdefault_valueif var is undefined, or blank${var-${nested}}- nested vars are supported up to 10 levels deep
The default_value can optionally be enclosed in single or double quotes. The quotes are necessary when the default value contains a closing brace }, e.g. ${var:-'{foo: bar}'}. The resulting value does not include the enclosing quotes.
Single quoted strings won't be interpolated, e.g.
key: '${this_will_not_be_interpolated}'
Note that default values within single quotes are still subject to interpolation, e.g.
variables:
foo: bar
things:
mqtt:topic:foo:
config:
payloadAvailable: "${available:-'${foo}'}"
# => because ${available} is undefined, it defaults to
# ${foo} which resolves to bar
Invalid syntax won't be interpolated, simply because the regex pattern won't match
key: ${vars can't have spaces}
$var syntax is not supported. The braces are mandatory, because I'm worried it might interfere with things like jsonpath patterns.
Escaping is current not supported: Plain text \${not_interpolated_because_its_escaped}.
Variable Scope / Precedence
- Variables declared in the including file (global) has precedence over variables declared inside the included file (local).
- Variables declared in the
!include { vars: { xxxx: } }overrides both global and local variables.
This is designed so that you can "include" a file and override its defaults / values and adjust it as needed.
Special Variables
__FILE__full absolute path of the current file, e.g./path/to/file.inc.yaml__FILE_NAME__only the filename portion without the extension or leading path, e.g.file.inc__FILE_EXT__only the extension portion, e.g.yaml__DIRECTORY__only the directory portion of the current file, e.g./path/to__DIR__alias for__DIRECTORY__
They can be accessed using the same variable syntax, i.e. ${__FILE_NAME__}
Including Other Files
Two syntaxes are supported for including other files as the "value" of a key
- Simple syntax:
keyname: !include <filename> - Full syntax:
keyname: !include
file: <filename>
vars:
varname: value
anothervar: anothervalue
# Or alternatively
keyname: !include { file: <filename>, vars: { varname: value, anothervar: anothervalue } }
Changes made to the included files will trigger an automatic reload of the main yaml file.
Variable interpolation is done on the full syntax, e.g.
keyname: !include
file: "${__FILE_NAME__}.inc.yaml" # needs to be double quoted
Include variable precedence
The variables defined in the vars key for the !include statement takes the highest priority over the toplevel variables.
main.yaml
variables:
var: toplevel
keyname: !include
file: subfile.inc.yaml
vars:
var: set_by_include
subfile.inc.yaml
variables:
var: locally_set
subkey: ${var} # => set_by_include
Packages
A Package file looks just like the main file except version: and readOnly: keys aren't needed and shouldn't be in it.
It can contain any number of valid elements (things, items, tags, and anything else that may be be implemented in the future).
Packages are simply merged into the main file into their corresponding top-level elements.
Package combined with variable substitutions allow the use of one package file into many actual instances because the variable substitutions make each inclusion to have unique IDs.
A package file can contain the full definition of a "device", which includes Things and Items related for that device. This can be "instantiated" by providing the package with specific device ID (thing id, item names) using include vars.
@lolodomo FYI.
This shouldn't conflict with anything that you're working on, I think.
rebased to main
Todo: merge lists in packages
@lolodomo When you find time, I would appreciate a short review from your end.
Great Work! I'll take a deeper look at this this week but some questions / remarks so I have a better understanding:
- Do per file variables need to be defined as global variables or is it sufficient to define them in the include directive?
- Shouldn't
__PATH__be named__FOLDER__,__DIRECTORY__or__PARENT__ Single quoted strings won't be interpolated,What's your reasoning or idea behind this?- What is returned for
${var:-'{}'}if var is undefined?
- Do per file variables need to be defined as global variables or is it sufficient to define them in the include directive?
No, they only need to be defined in the include directive.
- Shouldn't
__PATH__be named__FOLDER__,__DIRECTORY__or__PARENT__
Out of the three options, I prefer __DIRECTORY__. Is PATH too ambiguous?
Single quoted strings won't be interpolated,What's your reasoning or idea behind this?
So that you can have ${xxx} without being interpolated?
- What is returned for
${var:-'{}'}if var is undefined?
Good question. As it is now, it will return '{}' - however, it will trip up when you have ${var:-blah}${var:-blah} because the pattern for the "default" value is greedy.
To fix this, I need to make it ungreedy, but it would no longer work for ${var:-'{}'} case. I think this is an acceptable compromise without making the regex pattern more complex or implementing a separate parser. So the next change will have this requirement: The default value cannot contain } character.
@spacemanspiff2007 I added a commit that supports quoted defaults (in which you can use braces).
${var:-'{}'}=>{}when var is undefined
@jimtng I really like what you did in this PR! This is going to be a great feature which will help a lot and make configuration a breeze.
Is PATH too ambiguous?
In python there is a pathlib which deals with paths to dirs, files, drives, shares so PATH became synonymous to me with location of an arbitrary file system object. I think __DIRECTORY__ would be a nicer and also more correct.
Instead of packages how about we name the top level key templates or includes?
Package sounds like a couple of files yet it's only one file that's used.
Would it also be possible to not have to use the !include directive?
It's clear from the top level key that it's a reference to another file and what's actually happening is a merge and not an include.
E.g. like this:
packages:
livingroom-light1:
file: esphome-light.inc.yaml
vars:
thingid: livingroom-light1
How about we also add a templates? (or whatever name we settle for the top level key) and includes? subfolder to the YamlModelRepositoryImpl exclusion regex?
So that you can have
${xxx}without being interpolated?
I understand. I guess it's quite unexpected that " and ' quoted strings behave differently.
Maybe we can add the possibility to escape variable expansion with \${...} later or add a global option to not do the expansion for a file at all later. Then the behavior could be aligned which would be more user friendly.
@spacemanspiff2007
__PATH__ is now __DIRECTORY__ although I was tempted to shorten it to __DIR__. It was already done in a commit prior to your last message.
Instead of
packageshow about we name the top level keytemplatesorincludes? Package sounds like a couple of files yet it's only one file that's used.
templates is definitely not correct for it.
includes is awkward, IMO.
I think packages is a good name. The idea originally came from https://esphome.io/components/packages.html
Package sounds like a couple of files yet it's only one file that's used.
The packages section would contain a list of includes, so it does contain several files/packages
packages:
package1:
package2:
package3:
the package keys not being used bothers me a bit. Perhaps it should just be a list instead of a map, but I went with a map to make it all consistent with the rest of the convention for things, items, tags.
Perhaps I can pass an implicit variable ${PACKAGE_KEY} into the included file to make it useful.
Would it also be possible to not have to use the
!includedirective?
I'll consider it.
How about we also add a
templates?(or whatever name we settle for the top level key) andincludes?subfolder to the YamlModelRepositoryImpl exclusion regex?
I think with the exclusion of *.inc.yaml it should be enough. Adding more subdirectories to exclusion may potentially cause confusion.
You can of course structure your folder as ./includes/foo.inc.yaml if you wish.
So that you can have
${xxx}without being interpolated?I understand. I guess it's quite unexpected that
"and'quoted strings behave differently.
This is consistent with sh / bash / ruby, e.g.
$ A=test
$ echo "$A"
test
$ echo '$A'
$A
Maybe we can add the possibility to escape variable expansion with
\${...}later
I thought of this, but it's a can of worm that I'd rather not have to deal with yet.
__PATH__is now__DIRECTORY__although I was tempted to shorten it to__DIR__. It was already done in a commit prior to your last message.
I saw it after I sent the message. Both names are fine with me.
the package keys not being used bothers me a bit. Perhaps it should just be a list instead of a map, but I went with a map to make it all consistent with the rest of the convention for things, items, tags.
Perhaps I can pass an implicit variable
${PACKAGE_KEY}into the included file to make it useful.
For items and things the key is the unique identifier. Since these packages are fully merged configs they have no unique identifier (which would be the path to the file). If you find a good use case for the PACKAGE_KEY then a mapping is okay, otherwise I would prefer a list.
Will you provide support for the short syntax e.g. !include esphome-light.inc.yaml?
I'll consider it.
I thought some more and it might make sense to keep the !include directive.
File exclusion is done by matching the .inc.yaml so the !include should be kept since it aligns the the inc part in the filename.
I think
packagesis a good name. The idea originally came from https://esphome.io/components/packages.html
Yes - I noticed how everything is inspired by esphome.
However I'm not fully sold on packages, maybe just merge?
Since that's what it does - it merges the configs together.
Will a change in a package/included file trigger a reload of the referencing file?
the package keys not being used bothers me a bit. Perhaps it should just be a list instead of a map, but I went with a map to make it all consistent with the rest of the convention for things, items, tags. Perhaps I can pass an implicit variable
${PACKAGE_KEY}into the included file to make it useful.For items and things the key is the unique identifier. Since these packages are fully merged configs they have no unique identifier (which would be the path to the file). If you find a good use case for the
PACKAGE_KEYthen a mapping is okay, otherwise I would prefer a list.
It would be entirely up to the user to use it how they see fit. For me, I would use the PACKAGE_KEY as the thingid in the example above so there is no need to specify thingid vars.
Will you provide support for the short syntax e.g.
!include esphome-light.inc.yaml?
!include supports short syntax, so yes, that's automatically possible in the packages section too. However, a plain include like that isn't very useful because you could just as easily turn that into a full fledged yaml file on its own (with version: xx) and have it be loaded just like any other normal yaml file.
I thought some more and it might make sense to keep the
!includedirective. File exclusion is done by matching the.inc.yamlso the!includeshould be kept since it aligns the theincpart in the filename.
That is true, although that wasn't the main reason. Using !include is a "free" operation courtesy of the main/generic !include implementation for other purposes. All the packages section is concerned about is merely how to merge the (already included) packages.
I've just tried implementing what you suggested and it resulted in more complexity and code duplication. This is of course simply an implementation detail invisible to the user.
Will a change in a package/included file trigger a reload of the referencing file?
Not currently, however I did consider this possibility. There are pros and cons to doing this, and I opted not doing this, at least for now. The cons: sometimes I might want to make changes to several templates without having it reloaded, and then I can just trigger one reload of the main yaml file. But perhaps it's fine for it to reload several times too.
However I'm not fully sold on
packages, maybe justmerge?
I think packages is a more apt term. It means that you can package a device elements (things, items, custom tags) into one file.
merge is more "generic" in meaning, and it doesn't convey the meaning that it's a way to create a cohesive unit like the term package does.
Instead of
packageshow about we name the top level keytemplates
templates is a OH concept (not sure that it will ever end up in these configuration files, but still..) so I would leave that alone.
I guess it's quite unexpected that
"and'quoted strings behave differently.
I think it's rather common that these have different "rules" when it comes to substitutions, escaping etc. I'm not weighing in on whether it's a good idea here, I'm just saying that generally I think that is acceptable. But, doesn't YAML already have some "rules" regarding quoting to take into account? As far as I know, YAML only allows gives the escape character \ "special meaning" in double quotes, which means that you have to use double quotes if you want to include a character that needs escaping.
although I was tempted to shorten it to
__DIR__.
That's preferable IMO. When making such "special codes", you might as well make them easy to use.
but I went with a map to make it all consistent with the rest of the convention for things, items, tags.
Don't. It shouldn't become "a mantra" that is more important than reason. People will have to learn to use arrays too, and it's the only thing that makes sense here.
However I'm not fully sold on
packages, maybe justmerge?
In reality the terms matter less than we think, as soon as something is chosen and its meaning is well-defined and documented, people will quickly "get used to it". If the similarity with the ESPHome format is kept, parity with it might have a value in itself (for users using both systems).
!includesupports short syntax, so yes, that's automatically possible in the packages section too. However, a plain include like that isn't very useful because you could just as easily turn that into a full fledged yaml file on its own (withversion: xx) and have it be loaded just like any other normal yaml file.
Your right about the !include in packages.
But how about reusing parts of the configuration like this:
items:
my_item_name: !include energy_item.inc.yaml
my_other_name: !include energy_item.inc.yaml
energy_item.inc.yaml:
label: Energy
format: "%.1f"
unit: "kWh"
...
So this will be supported?
I've just tried implementing what you suggested and it resulted in more complexity and code duplication. This is of course simply an implementation detail invisible to the user.
Let's keep the !include then
Not currently, however I did consider this possibility. There are pros and cons to doing this, and I opted not doing this, at least for now. The cons: sometimes I might want to make changes to several templates without having it reloaded, and then I can just trigger one reload of the main yaml file. But perhaps it's fine for it to reload several times too.
But imho this would work the same way if you need to do changes now in an items and things file. You edit the files outside the folder and then move them into the openhab config folder. It's unexpected if some file edits will be picked up by openHAB and other won't.
If I reuse an .inc.yaml file in multiple files I have to remember which files to reload when I do a change.
Do you think it would be much effort to implement a reload when the include file changes?
Your right about the
!includein packages. But how about reusing parts of the configuration like this:items: my_item_name: !include energy_item.inc.yaml my_other_name: !include energy_item.inc.yaml
energy_item.inc.yaml:label: Energy format: "%.1f" unit: "kWh" ...So this will be supported?
I haven't tried this exact scenario but yes it should work just like you expected. That's essentially how !include works in all cases.
Not currently, however I did consider this possibility. There are pros and cons to doing this, and I opted not doing this, at least for now. The cons: sometimes I might want to make changes to several templates without having it reloaded, and then I can just trigger one reload of the main yaml file. But perhaps it's fine for it to reload several times too.
But imho this would work the same way if you need to do changes now in an items and things file. You edit the files outside the folder and then move them into the openhab config folder.
I think it depends. Besides, most people edit files in-place.
It's unexpected if some file edits will be picked up by openHAB and other won't.
The two files are different. One is an include file, the other is the main file.
If I reuse an
.inc.yamlfile in multiple files I have to remember which files to reload when I do a change.
Yes, exactly the idea, but I do see the convenience of having it auto reload and on the other hand, it can be annoying for those who like total control that eventually someone would ask for the option to turn it off. I am so used to doing it manually and the way I currently do it is using one big main file, so I only need to reload one file.
Do you think it would be much effort to implement a reload when the include file changes?
It would be an effort yes, and I am currently on vacation with very limited "computer" time until end of June. This will have to be a separate PR later.
but I went with a map to make it all consistent with the rest of the convention for things, items, tags.
Don't. It shouldn't become "a mantra" that is more important than reason. People will have to learn to use arrays too, and it's the only thing that makes sense here.
It's not merely following a mantra. I actually much prefer to have it as a map. But supporting a list is easy(easier), I'll see what it looks like and try it and if it looks OK, I'll consider supporting both syntaxes perhaps.
Yes, exactly the idea, but I do see the convenience of having it auto reload and on the other hand, it can be annoying for those who like total control that eventually someone would ask for the option to turn it off. I am so used to doing it manually and the way I currently do it is using one big main file, so I only need to reload one file.
I think the only sensible thing is that the templates "hot reload" too, otherwise you'd have to figure out and chase down which files are affected. I mean, the point of using a template is to avoid duplicating information. If you still have to hunt down the individual files and resave them, you could almost might as well just have copy/pasted.
But, I guess that this can be a challenge implementation-wise. You would have to keep track of basically all "YAML model" files, and then trigger reparsing when required. It sounds like a path full of pitfalls.
As I said, it depends on your workflow. I've been using a similar system and workflow in openhab for at least three years and preferred manual reloading.
I have an implementation in mind on how to do the auto reload but will also need to add a setting not to do it for those who prefer not having it. I also have in mind how to do this
A.global option through service config B. Granular option through include Param (least preferred)
I'll implement this in a late pr if/when time permits or at the end of June.
Everybody was tired today and returned early, so I had some time to implement the include dependency tracking.
Now whenever any include files are changed, the main yaml will be reloaded.
This pull request has been mentioned on openHAB Community. There might be relevant details there:
https://community.openhab.org/t/ideas-and-discussion-what-features-do-you-want-in-openhab-5-0/160573/757
@jimtng : is your initial message up-to-date with the current PR code? I try to understand it, it is very complex, even if you provide a big description.
@lolodomo yes, afaik, it should be up to date.
It basically has two parts: YamlPreprocessor.java contains the main logic. The other files / classes are just plumbings/implementations for Snakeyaml.
@jimtng I've just stumpled across hidden items in esphome. Do you think it makes sense to implement something like it in openHAB?
@jimtng I've just stumpled across hidden items in esphome. Do you think it makes sense to implement something like it in openHAB?
Sorry I didn't read the actual link!
The hidden items feature is interesting and could be useful.
I'd like to wait until this current PR is accepted before adding this hidden items feature. It doesn't seem too difficult to add, but it would probably be implemented a different section of the code.
Meanwhile, could you please open an issue for this so we don't forget?
Rebased to main
@lolodomo / @kaikreuzer will this one not make it to 5.0? Sorry to ask, but the code freeze deadline is near. If so, I'll need to remove it from the doc PR and hopefully we can merge the yaml doc for 5.0 release (without the extensions / enhancements).
If planning remains unchanged, there is no chance I can help reviewing this PR before 5.0 feature freeze. But I should be able to start having a look soon after 5.0 feature freeze. So this feature could be potentially included in very early 5.1 snapshots.
Thanks, I'll remove it from the docs for now.
Maybe we need to continue here to atleast get it in the next 5.1 milestone?
Thank you.Need help can you open this file
Sent from Yahoo Mail for iPhone
On Wednesday, August 13, 2025, 1:27 PM, lsiepel @.***> wrote:
lsiepel left a comment (openhab/openhab-core#4818) Maybe we need to continue here to atleast get it in the next 5.1 milestone?
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>
I’m old school how do I do that
Sent from Yahoo Mail for iPhone
On Wednesday, August 13, 2025, 4:06 PM, Krista Klugow @.***> wrote:
Thank you.Need help can you open this file
Sent from Yahoo Mail for iPhone
On Wednesday, August 13, 2025, 1:27 PM, lsiepel @.***> wrote:
lsiepel left a comment (openhab/openhab-core#4818) Maybe we need to continue here to atleast get it in the next 5.1 milestone?
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>