Device detection broken for colored lights

[agross]

ioBroker's Device Type Detector (UI) no longer detects colored lights as such after upgrading to version 2.x of this adapter.

Automatic detection is required for colored lights to be useful to the Lovelace adapter, which bundles state, color, brightness, and hue into a single entity.

Adapter version

2.0.5

[agross]

Also, due to upgrading, I lost about 90% of custom device names. To what I read on the forum this is due to dev_names.json containing almost no custom names. I wonder why.

{
  "0017880106fbbd3e": "Living Room Signe Floor",
  "0017880104990734": "Living Room KandyLight",
  "00178801044b8b37": "915005106701",
  "001788010407dc4d": "915005106701",
  "00178801082e99a8": "9290022166",
  "0017880108376b6c": "9290022166",
  "00178801061d8f64": "9290022166",
  "0017880106a0aaf5": "9290022166",
  "0017880108488e90": "5062431P7",
  "00178801084ceab3": "5062431P7",
  "00178801084ceb60": "5062431P7",
  "00178801084ce9b9": "5062431P7",
  "0017880108c48e3e": "5062231P7",
  "0017880108c4c8f2": "5062231P7",
  "00178801067ccd84": "Bedroom Signe Desk",
  "0017880106858c51": "3261030P7",
  "00178801093808a9": "9290022166",
  "0017880108ab6f69": "9290022166",
  "00158d000483b44d": "DJT11LM",
  "680ae2fffea72a25": "LED1732G11",
  "00158d00045c1216": "MCCGQ11LM",
  "00158d00045bedc5": "MCCGQ11LM",
  "588e81fffe2bacf4": "Kitchen Switch",
  "588e81fffe17a8ca": "E1743",
  "001788010872fbc4": "Living Room Hue Remote",
  "680ae2fffec237f9": "LED1732G11",
  "680ae2fffea24937": "LED1732G11",
  "588e81fffe15382a": "LED1732G11",
  "0017880108a658af": "9290022166",
  "0017880109c3ba2b": "1743130P7",
  "001788010e377c5a": "Bathroom",
  "00158d0009cbba65": "Outside Sensor",
  "588e81fffe2a6948": "Dining Switch",
  "group_901": "Group 901"
}

[asgothian]

Please attempt the following:

• stop the zigbee adapter
• rename the file LocalOverrides.json
• ensure that the dev_names.json is present
• restart.
• Check the log - it should either confirm reading the dev-names.json or complain that it cannot.

As far as the device detector is concerned - this is a side effect of cleaning up the color implementation. The bundling of 'hue' and 'color' is an anachronism, as those two may never be used together, and the use of 'hue' also requires the use of 'saturation'. for the detection to work.

I am not sure why the device detector does not detect the device with just state, brightness and color, as these have the same roles assigned as before.

Please provide a sample to an exact device as it was shown in the adapter before and after the change.

Also - please describe in detail how the device detector is used so this can be tested. Checking the 'required' vs 'optional' characteristics of color lamp devices shows that the states generated by the adapter do provide correct roles to all of the 'required' states in the device definition. It is possible that 'optional' states are no longer detected due to being hidden inside channels rather than as direct states on the device.

As I do not use Lovelace, or any automatic device detecting capabilities, I cannot test this.

A.

[agross]

please describe in detail how the device detector is used so this can be tested. I cannot test this.

Not sure what you mean by describing in detail. The test is whether RGB lights show up in the device detector UI (see below).

The device detector looks for patterns in states and determines if some collection of states could be considered to be a e.g. light. Lovelace then takes these detected devices and offers them to configure as entities. (@Garfonso: Correct?)

As an end-user you do not "use" the device detector yourself, despite a UI being available to check what was detected.

With 1.0.14 this list of lights on my ioBroker looks like this:

As of iobroker.zigbee 2.x this list no longer contains RGB lights, but only "white" lights.

Please provide a sample to an exact device as it was shown in the adapter before and after the change.

An example of a white light that still works on 2.x:

An example of an RGB light that is detected on 1.x only:

[asgothian]

again - I am not using the device detector. when I open the ui it shown no devices, neither white nor other lights, no matter the zigbee adapter version.

The entire tree for the 'native devices' is empty.

I did read through the devices.md, listing all the required roles for various devices - the required roles are present, so I cannot say why nothing is detected for me. But this makes it impossible for me to actually verify what is missing. Even going through the code you posted, I cannot find any reason why the devices are not detected - the required role is there (level.color.rgb), and all the other details are set too (type string, writeable, etc...)

You can attempt to return coloured lights to the 'old' representation without swapping back to the 1.10.14 adapter by adding them to the legacy override setting. If the device is defined within the zigbee Adapter, it should then use the old state names.

I have no use case for the devices or the UI, so I do not know where to start. I am interested in getting this sorted - but without assistance by people actually using this feature (and knowledgable on it), this seems very hard to do.

With the sample devices shown, please include the state descriptions for the recognised states as well, so I can take a look at any relevant differences.

A.

p.s. Note - as I have only zigbee-devices in use, I do not have the discovery adapter installed. There is no need to 'discover' any devices. I had to activate the checkbox on the devices tab in order to get the UI to show to begin with.

[agross]

This is the RGB light from above on 1.x:

I can't provide a screenshot from 2.x because I didn't take one. I will try to do the upgrade-screenshot-downgrade cycle tomorrow.

And yes, it's hard to debug the type/device detector. I've asked the maintainers if they can chime in here.

I am also not interested in detecting devices. The UI is just the easiest way to inspect the results of the type detector. At least this is my belief, I might be wrong.

[Apollon77]

@asgothian In fact it is all aboiut provideing the right state details like datatype, role read/write flags and then type detector works.

So in fact if detection worked before and now no longer this needs to be compared and likely the issue lies there and can be fixed by providing correct details in the state definitions. So if the above screenshot shows "zigbee 1.x" state definitions then we need another one with the "2.x" ones to compare.

But yes ideally the whole object definition should be checked. Admin just shows the role which is just one aspect

[asgothian]

@Apollon77 Thanks for the clarification. Now for the key question - how do I get the UI into my development system. It is not there, making it hard for me to test.

Also, will the introduction of channels in a device affect the device detector, if these also contain 'valid' roles:

Example:

A.

[Apollon77]

"development system" aka dev-server? Just install Devices adapter or which adapter ever is relevant ... but I am not sure if this will help too much beside seeing the detection results ... not sure if this will really help in understanding root causes of detection misses.

[asgothian]

No - just a normal installation for development purposes. I did not even try to attempt the dev-server. I'll give it a go with the devices adapter.

That worked fine. Now I have the issue that I still don't get any devices to show up despite having zigbee-devices with 'fitting' roles. So I seem to be lacking another piece for the puzzle

[Apollon77]

Maybe trx the atter adapter. There we use a different way where you can select a channel or device and it shows you which debvice types were discovered

[asgothian]

Interesting,

The matter adapter has no issues detecting the zigbee rgb lights, but does not map any hue/saturation command to the device.

It also doesn’t list any “automatically detected devices”

Ok - one bug found - the hue/saturation DP are classified wrongly. Now they are detected correctly too.

[asgothian]

I added an issue to the device detector project, as there is no viable solution for the Zigbee-Adapter to address this issue. Removal of color channels is not an option, as improper use of color states has caused numerous issues with the control of color lights. The channels were introduced to permanently fix this issue.

A solution without channels may be feasible, but is not currently being implemented in the adapter due to included issues with usability / readability for cases where the device-detector is not being used.

[asgothian]

After a detailed check, no changes on the zigbee adapter (aside of fixing the bug mentioned above) will be done. Recovery of the device detector is reliant on the fix of the issue linked above.

Note - the 'intended behaviour' label goes to the zigbee-adapter functionality, not to the loss of ability of using the device detection, but this needs to be fixed in the device detection, not the zigbee Adapter.

@agross - do existing / active devices in lovelace vanish when the new adapter is being used or is it just not possible to add new devices ?

A.

[agross]

You cannot use those lights that are no longer detected (either existing or new). Might not be a problem for Vis users, but for Lovelace detection is essential.

[asgothian]

As I wrote above - this needs to be fixed in the device detector, not the zigbee Adapter. The device detector does not behave as it says it will.

Also - this is by far not vital for Lovelace - It is still possible to generate Aliases for the Lights and use those. As a matter of fact - it is strongly suggested to not use the native zigbee hardware devices in Skripts or virtualisation, but always go through alias in order to be able to replace Hardware without having to change all the scripts / Visualisations.

A.

[agross]

I can grep/sed/awk my scripts for device IDs, no worries.

[asgothian]

You will have to remain with the adapter 1.10.14 until the Device-Detector is fixed to ignore channels, and until this change is integrated into the Lovelace adapter.

A.

[Apollon77]

Yes this detection isue is known and in progress on my side already ... but then also affected adapters needs updates.

So the "official workaround" would be to simply manuaky create aliasses to map it into inone channel ... so in theory no need to tweak adapters to support both tbh

[agross]

So the "official workaround" would be to simply manuaky create aliasses to map it into inone channel ... so in theory no need to tweak adapters to support both tbh

I don’t think it is feasible for most users to have them "simply manually" create aliases, let alone know that they would have to do it in the first place. Most users (including me) expect that adapters just work as in "I have lights and I want to control them with Lovelace". You need to know a lot of internals about ioBroker to know what to look for, for example that the type detector even exists and that Lovelace uses its outputs.

[asgothian]

The ‘official’ workaround is to remain with version 1.10.14 of the zigbee adapter until this is fixed.

[asgothian]

@Apollon77 Does it make sense to discuss ‘flagging’ states (and/or channels) with a ‘device’ status flag (possible values being ‘include’, ‘ignore’, ‘none’ or ‘operation’, ’setup’, ’management’) so an adapter can try to help with detection ? Does something like this exist already ?

A.

[asgothian]

@Apollon77 Anbei die erfragten JSON Dateien: 5 Geräte:

• zigbee.0.0017880103124d6d-2 - Philips Hue Leuchtmittel mit Color_HS, Color_xy, Color_rgb, Colortemp
• zigbee.0.847127fffe9a7fd6-2 - Lidl Lightbar mit Color_HS und colortemp, aber ohne Color_rgb (mit getrennten R, G, B). Wichtig: diese Leuchte reagiert trotzdem korrekt auf 'benannte Farben' (rot, gelb, etc.) und 6digit hex werte (ohne # - ff0000, 00abcd, etc)
• zigbee.0.60a423fffe6f1e83-2 - Lidl Leuchtmittel mit colortemp, aber ohne color
• zigbee.0.0017880100ed235c - Ikea Leuchtmittel ohne colortemp, ohne color
• zigbee.0.8cf681fffeee45fc - Leuchtmittel mit Color_rgb und colortemp, aber ohne color_hs

zigbee.0.8cf681fffeee45fc.json zigbee.0.0017880100ed235c.json zigbee.0.60a423fffe6f1e83-2.json zigbee.0.847127fffe9a7fd6-2.json zigbee.0.0017880103124d6d-2.json

[Apollon77]

Ok, I will use this also as test cases but one thing: The Type detector will not detect multiple types at once or it might be strange. So likely it will detect an "RGB" device as soon as separate R,G,B states and pot level, temperature and switch is there. So, also if xy or HS states are also there they would be ignored. or being detected as an own device . So in fact just "because the device has the stuff or you can map it somehow" it might not make sense to just add all the states. This might just confuse type detector (and also other mechanisms).

So I tested e.g. the zigbee.0.0017880103124d6d-2 is detected as:

• Main -> RGB with R,G,B, Dimmer, Temperature, On-State
• 2. RGBSingle -> just zigbee.0.AAAAAAA.color state
• 3. Hue -> Hue and Saturation
• 4.-6 ignore because tries tu guess from the rest

But in fact it is "fine" because all currently logics just take the first detected entry as the best one (which is also the case here likely). I proposed now to adjust the detection order and favor HS before RGB which would lead to the fact that it would detect HS instead or RGB as first

Notes: ioBoker do not support separated x/y states ... the definition for CIE is to have a combined array like state with [x,y] as (stringified array) value. That means the the separated x/y states will never be mapped to a CIE light!

[asgothian]

Ok, I will use this also as test cases but one thing: The Type detector will not detect multiple types at once or it might be strange. So likely it will detect an "RGB" device as soon as separate R,G,B states and pot level, temperature and switch is there. So, also if xy or HS states are also there they would be ignored. or being detected as an own device . So in fact just "because the device has the stuff or you can map it somehow" it might not make sense to just add all the states. This might just confuse type detector (and also other mechanisms).

So I tested e.g. the zigbee.0.0017880103124d6d-2 is detected as:

• Main -> RGB with R,G,B, Dimmer, Temperature, On-State
• 2. RGBSingle -> just zigbee.0.AAAAAAA.color state
• 3. Hue -> Hue and Saturation
• 4.-6 ignore because tries tu guess from the rest That is fine by me.

Questions

• the RGB with R,G,B is using the sub channels for R, G, B with level.color.r, level.color.g, level.color.b or the combined with level.color.rgb
• What will happen if a sub channel includes no valid role (i.e. just role = state) ? Can the zigbee Adapter 'guide' the type detector towards R,G,B, RGBSingle, HS, CIE by selectively assigning the respective roles ?
• what if I add a state in a channel with a level.color.rgb role, and at the same time remove the level.color.rgb role from the zigbee.0.xxxx.color state ? (Reason for this is that all the channel methods feed into this state, so the value in this state could be:
• aabbcc
• {"r":123,"g":11, "b":231}
• red
• {"hue":123, "saturation":90},

As only one of them follows the expectation for an RGBSingle light, it may not be the best state for this role.

But in fact it is "fine" because all currently logics just take the first detected entry as the best one (which is also the case here likely). I proposed now to adjust the detection order and favor HS before RGB which would lead to the fact that it would detect HS instead or RGB as first

I am fine with this. I am also willing to generate a setting in the zigbee Adapter to allow the user to 'order' the device detection priority between HS, R,G,B, RGBSingle, CIE (see below) - if I can control this simply by setting the respective roles depending on what is supported.

Notes: ioBoker do not support separated x/y states ... the definition for CIE is to have a combined array like state with [x,y] as (stringified array) value. That means the the separated x/y states will never be mapped to a CIE light!

As far as I know this should not pose an issue. I will do a few tests and possibly generate a respective state which does expect a [x,y] array - parsing this into the correct format for the zigbee communication ?

Please note that from my perspective it is very important that the device detector learns to ignore channels which contain no states with 'valid' roles for detection.

[Apollon77]

Hey,

the RGB with R,G,B is using the sub channels for R, G, B with level.color.r, level.color.g, level.color.b or the combined with level.color.rgb

The current order will detect the "3 state" version over the "One state"version.

What will happen if a sub channel includes no valid role (i.e. just role = state) ? Can the zigbee Adapter 'guide' the type detector towards R,G,B, RGBSingle, HS, CIE by selectively assigning the respective roles ?

In fact if roles are missing type detector will likely ignore things, so yes in theory the adapter so could guide it ... BUT we will add now options so that visus and "consuming adapters" can also prioritize types, so matter would choose HUE/SAT before RGB as example because it would be more natural ... while web visus might choose RGB because colorpicker support that better ... so I think it is better to have roles correctly defined for the states that are available. The honest question is if you really need to map many other fields if the device does not have them generically ... yes it is user convenience but e.g. RGB->HSB/V always has pitfalls

level.color.rgb The value format is defined! It is #rrggbb or ##rrggbbww ... if you do not have this value then the role should not be used. It might be ok to accept other values (but also that could be confusing), but the value set by the adapter needs to be in the said format if the role is that way. And Honestly: I would (for users and also me as dev) favor a "one state" approach" rather than a multi state approach if possible because setting that is atomic :-) So question is ... why you offer rgb separated states at all? :-)

I am fine with this. I am also willing to generate a setting in the zigbee Adapter to allow the user to 'order' the device detection priority between HS, R,G,B, RGBSingle, CIE (see below)

See above ... that might be no good idea because that could limit certain visus from seeing it. In my eyes place the states that make sense, and if multiple then also ok, because it is in fact up to the visu/consuming adapter what he can best use.

Please note that from my perspective it is very important that the device detector learns to ignore channels which contain no states with 'valid' roles for detection.

He does that already. Or do you have examples?

[asgothian]

Hey,

What will happen if a sub channel includes no valid role (i.e. just role = state) ? Can the zigbee Adapter 'guide' the type detector towards R,G,B, RGBSingle, HS, CIE by selectively assigning the respective roles ?

In fact if roles are missing type detector will likely ignore things, so yes in theory the adapter so could guide it ... BUT we will add now options so that visus and "consuming adapters" can also prioritize types, so matter would choose HUE/SAT before RGB as example because it would be more natural ... while web visus might choose RGB because colorpicker support that better ... so I think it is better to have roles correctly defined for the states that are available. The honest question is if you really need to map many other fields if the device does not have them generically ... yes it is user convenience but e.g. RGB->HSB/V always has pitfalls

I can go with that, and will offer all available states with correct roles set. Easier for me.

level.color.rgb

The value format is defined! It is #rrggbb or ##rrggbbww ... if you do not have this value then the role should not be used. It might be ok to accept other values (but also that could be confusing), but the value set by the adapter needs to be in the said format if the role is that way. And Honestly: I would (for users and also me as dev) favor a "one state" approach" rather than a multi state approach if possible because setting that is atomic :-) So question is ... why you offer rgb separated states at all? :-)

The answer is a little tricky: The ZHC color converter will accept several of the above forms of defining color as payload. Since we currently match converter <-> state, the converter leads to the state color, but still accepts a number of those formats into the same state.

Without any additional states, the device detector would see only one state which doesn't actually work properly for the defined role. Previously, the adapter would also generate hue, saturation, r, g and b states (if applicable), but one as 'option', and the others as 'state', meaning the 'options' needs to be changed before the 'state' is, as the payload is built when the 'state' is changed and it includes the respective 'option'. There also was an implementation which would always send all connected values when one was changed, but that also was a messy solution, as conflicting payloads would be sent in quick succession. The key solution is to delay sending data for a while, so the both states can be updated before they are combined into a payload and sent. But this was/is unintuitive for the user, as they do not know which states are 'with delay' and which are 'immediate'. Hence the channels. All states within one channel are being collected into one payload - with a default 500 ms timeout (I am considering to lower this to 100), which is increased to 5s if the change is done via the admin (manually by a user). it is transparent to the user, as the color state changes when the data is being sent. Any change to the color state will still be sent immediately. When I set this up, I was unsure if there were any use cases where separate r,g,b values were beneficial. But as the option was readily available (in role definition, and for the zigbee payload) I decided to include them (same with the x,y ones, which will be changed to a color_cie state which expects [x,y] in the future.

Also - I am not certain if ##rrggbbww is a valid zigbee payload, as I do not have any zigbee device with this capability. I did check if I could find any device which uses this kind of payload, but did not find any.

He does that already. Or do you have examples?

I can generate an example for tomorrow. I know that the detector did not in fact ignore channels without valid roles when this issue was raised. I did test this. The existence of channels within the device threw the detector for a loop.

See above ... that might be no good idea because that could limit certain visus from seeing it. In my eyes place the states that make sense, and if multiple then also ok, because it is in fact up to the visu/consuming adapter what he can best use.

Fine by me - less work for me is always a good thing :)

With the adapter version 3.0.2, there will be 3 changes regarding the color:

• the current color state will remain as is, but will have its role set to state
• a separate rgbcolor state will be generated, which only accepts #rrggbb and which will carry the level.color.rgb role. This can be placed within a channel or flat next to the color state - I am fine with either in this case, as it will always be set 'as one'. If a zigbee-device with ##rrggbbww support surfaces, the support for that format will be in form of another additional state rgbwcolor which only accepts ##rrggbbww, and which will carry the correct role for this (I assume it is level.color.rgbw ?
• the color_xy channel will have its states x and y removed. Instead, there will be a state xy with the role level.color.cie and the expected value of [x,y]

Thoughts on this ?

A.

[Garfonso]

• a separate rgbcolor state will be generated, which only accepts #rrggbb and which will carry the level.color.rgb role. This can be placed within a channel or flat next to the color state - I am fine with either in this case, as it will always be set 'as one'. If a zigbee-device with ##rrggbbww support surfaces, the support for that format will be in form of another additional state rgbwcolor which only accepts ##rrggbbww, and which will carry the correct role for this (I assume it is level.color.rgbw ?

I won't put that one into its own channel. As it does not need to be delayed, it would clash with your concept above.

Maybe even think about moving the xy state back to the device from its channel.

All in all sounds like good changes. No matter if in channel or not. 😄

[Apollon77]

I will release matter 4.5.0 which includes the "Latest" type detector 4.5.0 later (latest repo will be delayed a bit, because we test first but can be manually installed). Would be cool to get infos how the "Mass detection" results are there for the new zigbee structures.

[asgothian]

I’ll run tests with matter 4.5 if I can.

[agross]

I somehow got @iobroker/type-detector 4.5.0 on my system, and lights are detected correctly with 3.0.1 (like before with 1.10.14).