[Feature request]: Additional workarounds to unfreeze IKEA lights

[lbschenkel]

Is your feature request related to a problem? Please describe

IKEA trådfri lights have a firmware bug in which if they are in the middle of a color/temperature transition, they "freeze" and will not respond to any other Zigbee commands while the transition is ongoing.

An example:

1. set light to 2000K
2. set light to 4000K with a transition of 10s
3. during these 10s the light will stop processing commands (for example: turning off the light)
4. Z2M light state becomes out of sync with the actual light state (for example: Z2M will report light off while the light is still on)
5. after 10s light completes the transition and starts responding to new commands
6. Z2M light state will stay out of sync

Notes:

• This also happens during scene transitions, if they trigger a color/temperature change.
• This does not happen for brightness transitions, only color/temperature.
• The light may start responding sooner if it has reached the target color/temperature before the transition time elapses (if it was already at the target color/temperature)

This bug may result in very undesirable behavior. A smart switch may be unable to turn off the light, for example, if the light is in the middle of a long transition (wake up light, flux / circadian / adaptive lighting, etc.) while Z2M will optimistically report that the light is off.

Extreme example: imagine a long transition, such as 30m: the light will simply freeze for the whole duration and you can't even turn it off, except by cutting the power.

Workaround

There is actually a way to "unfreeze" the light while it is transitioning: sending a color_temp_move: stop command (lightingColorCtrl.moveColorTemp: 0 at the Zigbee layer). This stops the color/temperature transition and consistently makes the light start responding again.

Describe the solution you'd like

Best option

Ideally Z2M would implement the following workaround in herdsman-converters for IKEA lights:

1. it notices that it is about to send a command that will (or could) change the light color/temperature
2. it looks at the transition time for this command, and remembers that the light will potentially be frozen until now + transition (let's assume that the time is stored in a per-light state called frozen_until)
3. when a new command needs to be sent to the light, it checks if the light is frozen (frozen_util is set and if now <= frozen_until). if it is: a. it sends lightingColorCtrl.moveColorTemp: 0 to unfreeze the light b. clears frozen_until c. then it sends the new command (by going back to step 1, which might set frozen_until again)

This implies that any new command sent to the light will make it stop transitioning. But I argue that this is much better than the light just silently ignoring all commands, potentially refusing to turn off, and causing the state to get out of sync.

For groups I imagine that the same would apply, as long as one ore more IKEA lights are members of the group. However, setting frozen_until in the group would also set it under the individual IKEA lights, so subsequent individual commands to the lights would still successfully unfroze them.

I'm not a fan of premature generalization, but if it must then perhaps it could be generalized for all devices in the following way:

• all devices have a frozen_until state
• when sending any Zigbee commands to a device, first check frozen_until and if applicable, send an unfreeze() command before sending the actual command
• only IKEA converters will set frozen_until, and only when they detect a potential color/brightness change with a transition
• only IKEA converters will override unfreeze() to send lightingColorCtrl.moveColorTemp: 0, others do nothing

Alternative

A simpler alternative could be simply to send an unconditional lightingColorCtrl.moveColorTemp: 0 command at critical points for IKEA lights, at minimum for on/off to make sure that the light can always be turned on/off even if it is frozen. But that will not be as nearly as effective, and would also generate more redundant Zigbee traffic in the network.

Describe alternatives you've considered

I have done the above via custom scripts that I wrote in Home Assistant. But it's a very awkward solution that requires my scripts to be called instead of the regular light.turn_on/turn_off services, or wrapping the actual Z2M light under a template light.

Naturally these workarounds won't even apply when not using Home Assistant. Doing this in Z2M is a much cleaner solution which benefits everybody.

Additional context

This is a well-known problem with IKEA lights:

• https://github.com/Koenkk/zigbee2mqtt/issues/1810
• https://github.com/Koenkk/zigbee2mqtt/issues/4316
• https://github.com/dresden-elektronik/deconz-rest-plugin/issues/894
• https://www.reddit.com/r/tradfri/comments/au903n/firmware_bugs_in_ikea_bulbs/

[Koenkk]

Nice analysis! Before fixing something, I first want to see how many people are affected by this (so please, if you are affected please put a 👍 on the OP)

[lbschenkel]

A correction: it's better to use brightness_move: stop than color_temp_move: stop to unfreeze the lights. Both work but the latter is not recognized by RGB lights (they don't support color temperature). I wrote the description from memory but I re-checked my scripts and I'm actually sending the former.

[lbschenkel]

@Koenkk: My knowledge of JS and the Z2M codebase is limited, but I was able to patch the source code to do a PoC of what I'm proposing and documenting what I could figure out with my IKEA lights.

First I've done it directly in the converters to check that it works but then I refactored it so I can run it in "production" via an external converter without touching anything else. The result is a bit crude and not very elegant and requires a lot of decoration, but it does work and it's the best way I could find to illustrate the logic. Naturally a "proper" solution will look different. My gut feeling is that putting this logic in the generic command code path and opted in by options in the device declaration in module exports should be very clean.

I also think this should be gated by an user-facing setting (as color_sync or state_action) , so users can opt out or opt in to this behavior. Not sure which default is best, but for sure I know what I will use :-).

Anyway, I have this in my house and it is working beautifully so far. I will report any issues that I encounter.

Things I haven't explored so far:

• how to do it for groups
• what should we do when there's no transition and light uses its "firmware built-in transition" (should we capture what that is for each light, or maybe be pessimistic and assume 1s)?
• scenes, I suppose we should just be pessimistic and assume that any scene+transition can freeze the light?

////////////////////////////////////////////////////////////////////////////////
// copied from ikea.js:

const extend = require('zigbee-herdsman-converters/lib/extend');
const globalStore = require('zigbee-herdsman-converters/lib/store');
const ota = require('zigbee-herdsman-converters/lib/ota');
const tz = require('zigbee-herdsman-converters/converters/toZigbee');
const utils = require('zigbee-herdsman-converters/lib/utils');

const bulbOnEvent = async (type, data, device, options, state) => {
    /**
     * IKEA bulbs lose their configured reportings when losing power.
     * A deviceAnnounce indicates they are powered on again.
     * Reconfigure the configured reoprting here.
     *
     * Additionally some other information is lost like
     *   color_options.execute_if_off. We also restore these.
     *
     * NOTE: binds are not lost so rebinding is not needed!
     */
    if (type === 'deviceAnnounce') {
        for (const endpoint of device.endpoints) {
            for (const c of endpoint.configuredReportings) {
                await endpoint.configureReporting(c.cluster.name, [{
                    attribute: c.attribute.name, minimumReportInterval: c.minimumReportInterval,
                    maximumReportInterval: c.maximumReportInterval, reportableChange: c.reportableChange,
                }]);
            }
        }

        // NOTE: execute_if_off default is false
        //       we only restore if true, to save unneeded network writes
        if (state !== undefined && state.color_options !== undefined && state.color_options.execute_if_off === true) {
            device.endpoints[0].write('lightingColorCtrl', {'options': 1});
        }
        if (state !== undefined && state.level_config !== undefined && state.level_config.execute_if_off === true) {
            device.endpoints[0].write('genLevelCtrl', {'options': 1});
        }
        if (state !== undefined && state.level_config !== undefined && state.level_config.on_level !== undefined) {
            let onLevel = state.level_config.on_level;
            if (typeof onLevel === 'string' && onLevel.toLowerCase() == 'previous') {
                onLevel = 255;
            } else {
                onLevel = Number(onLevel);
            }
            if (onLevel > 255) onLevel = 254;
            if (onLevel < 1) onLevel = 1;

            device.endpoints[0].write('genLevelCtrl', {onLevel});
        }
    }
};

const tradfriExtend = {
    light_onoff_brightness: (options = {}) => ({
        ...extend.light_onoff_brightness(options),
        ota: ota.tradfri,
        onEvent: bulbOnEvent,
    }),
    light_onoff_brightness_colortemp: (options = {colorTempRange: [250, 454]}) => ({
        ...extend.light_onoff_brightness_colortemp(options),
        ota: ota.tradfri,
        onEvent: bulbOnEvent,
    }),
    light_onoff_brightness_colortemp_color: (options = {disableColorTempStartup: true, colorTempRange: [250, 454]}) => ({
        ...extend.light_onoff_brightness_colortemp_color(options),
        ota: ota.tradfri,
        onEvent: bulbOnEvent,
    }),
    light_onoff_brightness_color: (options = {}) => ({
        ...extend.light_onoff_brightness_color(options),
        ota: ota.tradfri,
        onEvent: bulbOnEvent,
    }),
};

////////////////////////////////////////////////////////////////////////////////
/// unfreeze code:

const unfreezeMechanisms = {
    // WS lights:
    //   Aborts the color transition midway: light will stay at the intermediary
    //   state it was when it received the command.
    // Color lights:
    //   Do not support this command. 
    moveColorTemp : function() {
        this.entity.command('lightingColorCtrl', 'moveColorTemp', { rate: 1, movemode: 0, minimum: 0, maximum: 600 }, {});
    },

    // WS lights:
    //   Same as 'moveColorTemp'.
    // Color lights:
    //   Finishes the color transition instantly: light will instantly
    //   "fast forward" to the final state, post-transition.
    genLevelCtrl : function() {
        this.entity.command('genLevelCtrl', 'stop', {}, {});
    }
}

class UnfreezeSupport {
    
    #entity;
    #unfreeze;

    constructor (entity, mechanism) {
        this.entity = entity;
        this.unfreeze = mechanism;
    }
    
    willFreeze(clusterKey, commandKey, payload) {
        return payload 
            // any color command with a transition will freeze the light...
            && payload.transtime > 0
            && clusterKey == 'lightingColorCtrl'
            // ...except for 'stop' commands:
            && payload.rate != 1
            && payload.movemode != 0;
    }

    async command(clusterKey, commandKey, payload, options) {
        const frozenUtil = globalStore.getValue(this.entity, 'frozenUntil');
        if (frozenUtil != null) {
            if(Date.now() <= frozenUtil) {
                console.log("Light is frozen, will attempt to unfreeze");
                this.unfreeze();
            }
            globalStore.clearValue(this.entity, 'frozenUntil');
        }
        const returnValue = await this.entity.command(clusterKey, commandKey, payload, options);
        if (this.willFreeze(clusterKey, commandKey, payload)) {
            const millis = payload.transtime * 100;
            const frozenUntil = Date.now() + millis;
            globalStore.putValue(this.entity, 'frozenUntil', frozenUntil);
            console.log("Command", clusterKey + "." + commandKey, payload, options, "will freeze the light until", frozenUntil);

        }
        return returnValue;
    }
}

function unfreezeSupport (converters, mechanism) {
    const newConverters = new Array();
    for (const converter of converters) {
        const newConverter = {
            ...converter,
            convertSet: async (entity, key, value, meta) => {
                const unfreeze = new UnfreezeSupport(entity, mechanism);

                const newEntity = Object.create(entity);
                newEntity.command = async (clusterKey, commandKey, payload, options) => {
                    return await unfreeze.command(clusterKey, commandKey, payload, options);
                }
                
                return await converter.convertSet(newEntity, key, value, meta);
            }
        }
        newConverters.push(newConverter);
    }
    return newConverters;
}

module.exports = [
    {
        zigbeeModel: ['TRADFRI bulb E27 CWS opal 600lm', 'TRADFRI bulb E26 CWS opal 600lm', 'TRADFRI bulb E14 CWS opal 600lm',
            'TRADFRI bulb E12 CWS opal 600lm', 'TRADFRI bulb E27 C/WS opal 600'],
        model: 'LED1624G9',
        vendor: 'IKEA',
        description: 'TRADFRI LED bulb E14/E26/E27 600 lumen, dimmable, color, opal white',
        extend: tradfriExtend.light_onoff_brightness_colortemp_color(),
        toZigbee: unfreezeSupport(
            utils.replaceInArray(tradfriExtend.light_onoff_brightness_colortemp_color().toZigbee, [tz.light_color_colortemp], [tz.light_color_and_colortemp_via_color]),
            unfreezeMechanisms.genLevelCtrl), // moveColorTemp does not work for this light
        meta: { supportsHueAndSaturation: false },
    },
    {
        zigbeeModel: ['TRADFRI bulb E27 WS opal 1000lm', 'TRADFRI bulb E26 WS opal 1000lm'],
        model: 'LED1732G11',
        vendor: 'IKEA',
        description: 'TRADFRI LED bulb E27 1000 lumen, dimmable, white spectrum, opal white',
        extend: tradfriExtend.light_onoff_brightness_colortemp({ colorTempRange: [250, 454] }),
        toZigbee: unfreezeSupport(tradfriExtend.light_onoff_brightness_colortemp({ colorTempRange: [250, 454] }).toZigbee, unfreezeMechanisms.genLevelCtrl)
    },
    {
        zigbeeModel: ['TRADFRI bulb E27 WS opal 980lm', 'TRADFRI bulb E26 WS opal 980lm', 'TRADFRI bulb E27 WS\uFFFDopal 980lm'],
        model: 'LED1545G12',
        vendor: 'IKEA',
        description: 'TRADFRI LED bulb E26/E27 980 lumen, dimmable, white spectrum, opal white',
        extend: tradfriExtend.light_onoff_brightness_colortemp(),
        toZigbee: unfreezeSupport(tradfriExtend.light_onoff_brightness_colortemp().toZigbee, unfreezeMechanisms.genLevelCtrl)
    },
    {
        zigbeeModel: ['TRADFRI bulb GU10 WS 400lm'],
        model: 'LED1537R6/LED1739R5',
        vendor: 'IKEA',
        description: 'TRADFRI LED bulb GU10 400 lumen, dimmable, white spectrum',
        extend: tradfriExtend.light_onoff_brightness_colortemp(),
        toZigbee: unfreezeSupport(tradfriExtend.light_onoff_brightness_colortemp().toZigbee, unfreezeMechanisms.genLevelCtrl)
    }
];

[lbschenkel]

This is a crude test case:

#!/bin/bash

pub() {
    mosquitto_pub -h HOST -u USER -P PASSWORD \
    -t zigbee2mqtt/LIGHT/set -m "$1"
}

pub '{"state": "on", "brightness": 2, "color_temp": 150, "transition": 0}'
sleep 2
pub '{"brightness": 255, "color_temp": 500, "transition": 5}'
sleep 3
pub '{"state": "off"}'

Without the unfreeze code, the light is frozen for 5s and will ignore any commands in that timeframe -- so it will not actually turn off even though Z2M thinks it did and updates the state. With the unfreeze code, the light will stop transitioning and turn off.

1. Code won't assume that the light will freeze because transition is 0
2. Code will detect that the light will freeze (changing color with a non-zero transition)
3. Code will detect that the light is frozen and unfreeze the light before sending the actual command

[lbschenkel]

An update: I realized that I've made a mistake when evaluating the outcome of the "unfreezing mechanisms". All lights support lightingColorCtrl.moveColorTemp: 0, including the color lights that do not natively support temperature. What was actually producing the error I've seen was this read here: https://github.com/Koenkk/zigbee-herdsman-converters/blob/3a948abc5cc94c0a4be2ede464834431d4cd4148/src/converters/toZigbee.js#L762, which was happening with an early implementation of my workaround.

So, as far as I can see, both commands are equivalent and will successfully unfreeze any IKEA light that I have (a mixture of GU10/E14/E27 WS and color):

• lightingColorCtrl.moveColorTemp: 0
• genLevelCtrl.stop

[protyposis]

Fixing this would be advantageous for users of Adaptive Lighting, which uses very long color transitions, e.g., 45 seconds. The currently recommended workaround for long color transitions on Ikea bulbs is basically simulating it by "flooding" the Zigbee network with sequences of frequent instant color changes (https://github.com/basnijholt/adaptive-lighting#bulb-bulb-specific-issues).

[nstadigs]

Fixing this would be advantageous for users of Adaptive Lighting, which uses very long color transitions, e.g., 45 seconds. The currently recommended workaround for long color transitions on Ikea bulbs is basically simulating it by "flooding" the Zigbee network with sequences of frequent instant color changes (https://github.com/basnijholt/adaptive-lighting#bulb-bulb-specific-issues).

Maybe the opt-in option is preferred then?

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

[lbschenkel]

Please don't close this; @Koenkk what are your thoughts on the above?

[Koenkk]

given the thumbs up on the OP, more people are interested in this. Could you make a PR? Then we can iterate from there.

[lbschenkel]

All right, I will create a proper PR when able (but not with the above code, which was a roundabout way designed to work purely as an external converter to allow others to test it).

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

[francisp2]

?

[PetziAt]

any news on this topic? I'm in favour ;)

[lbschenkel]

I am going to submit a PR, but haven't had the time yet. Some other things came up that made me busy. But it'll come.

[boesing]

I am kinda affected. I did try to get color temp + brightness be modified with transitions (2 sec) but it seems that sometimes, only temperature is applied while brightness remains the old value. I do set 3 values when modifying:

1. transition time (2 sec)
2. color temp value
3. brightness level

I already realized that this happens and I tried to workaround with node-red "repeat message" where I re-send all 3 values in total for 4 times with 3 seconds delay but that does not change anything. Sometimes it works but most of the time, it does not.

[lbschenkel]

I already realized that this happens and I tried to workaround with node-red "repeat message" where I re-send all 3 values in total for 4 times with 3 seconds delay but that does not change anything. Sometimes it works but most of the time, it does not.

It will never work when you send all values at the same time. The lights don't support that, actually. What really happens is that Z2M splits your request into two Zigbee messages: (1) it sets the brightness level first, without any transition (the lack of transition here is an explicit IKEA workaround), and then (2) sets the color temperature with the transition. During (2) the lights will be "frozen".

If you change your automation to set only the brightness first with transition/2, then you wait for that time to elapse (+ 200 ms, which I found by experiment to be the time that lets the firmware "settle" between commands), then you set the color with transition/2 (keeping in mind that light will be frozen during this period), it should consistently "work". At least it does for me.

Without that 200ms in between I experienced many times that either the brightness change is "cancelled" and returns to the previous value or the 2nd command is ignored. Firmware in these IKEA lights really sucks.

[github-actions[bot]]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 30 days