WLED presets.cfg corrupted after a while

trafficstars

What happened?

after using the esp32 some days or weeks, the presets.cfg is corrupted - here is the part read from ../edit:

               "stop": 0
            }
        ]
    }
    se "                        ÿ   b                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        ÿÿÿÿÿÿ            ÿ                                   "
}, "col": [[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]], "fx": 115, "sx": 0, "ix": 79, "c1x": 128, "c2x": 128, "c3x": 128, "pal": 15, "sel": true, "rev": fÿlse, "rev2D": false, "mi": true, "rot2D": false
}, {
    "id::1," start ":17,[stop": 78,
    "grp": 1,
    "spc": 0,

see the special characters inbetween which caused the controller to declare is unreadable. After fixing it with a valid presets.cfg this happens again after some days or weeks.

It never happens with the cfg file afaik.

I have this issue not on every controller - but some of them (2-3). I did flashed the new with full erase, but that didn't help. Any ideas how to fix ? Could it be a hardware issue of the memory? And whre can i acces the json file which is shown as last working presets file ?

To Reproduce Bug

difficult - just happens by time to time after days or hours

Expected Behavior

readable json

Install Method

Binary from WLED.me

What version of WLED?

WLED 0.13b6 and 14.0

Which microcontroller/board are you seeing the problem on?

ESP32

Relevant log/trace output

No response

Anything else?

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Sep 22 '22 11:09 DarthWeber

The race condition has been fixed in 0.13.3. There were reports of this issue re-occurring on 0.13.3 though. In such case it may not be an issue of WLED but underlying LittleFS library or flash memory.

Sep 22 '22 12:09 blazoncek

Now its even worse: it is impoosible to restore ANY presets - the result is always: "Sorry, there was an issue...". I tried several presets.json even from some months ago, the result is always the same.

In the json editor i can not see any mistakes and i am very sure, that these json files worked for a restore already. It semme also be impossible to create new presets.

I also tried an update to 13.3 - but the problem remains. The restore of the cfg file has also some problems since the gpio pins and number of LEDs are also not restored.

Sep 22 '22 19:09 DarthWeber

Have you tried erasing flash? Use esptool when doing so.

Sep 23 '22 05:09 blazoncek

Just tried erasing the flash - and also tried two different fresh esp32. Always the same issue. I also checked the 14b0, it has also this problem. Here ist my presets.cfg - would be great if you could give it a try. My preset is always unreadable after i delete preset 99 or preset 12

wled-WC.zip

Sep 26 '22 14:09 DarthWeber

Now its even worse: it is impoosible to restore ANY presets - the result is always: "Sorry, there was an issue...". I tried several presets.json even from some months ago, the result is always the same.

Steps I have used to recover from this condition:

Check your backup here and make sure it is properly formatted.
Create an empty presets.json file and upload, then power cycle the controller
Restore known good backup file

In general, I am experiencing the same issue, including controllers running the latest 0.13.3 ... always an ESP32 controller, my esp8266 are not having this issue. I have also experienced the issue under previous versions.

Sep 29 '22 01:09 gordonthree

Having this issue as well. Seems to get corrupted by the esp32 and not from something I do since many times that this happens I am not changing any presets.

Sep 29 '22 02:09 craigthompson

This is a known issue and it mainly affects ESP32. It is most likely caused by dual core nature of ESP32 and the way LittleFS and AsyncWebServer libraries are written. There is little we can currently do about it but we are investigating other options how to write data to presets.

If you can reliably reproduce corruption, please provide steps to reproduce it on independent system.

Sep 29 '22 05:09 blazoncek

Reliably reproducing seems to be VERY difficult. I have this issue often with controllers that are switched off at night but maybe this has nothing to do with it. Another reason could be that a write a preset to store the actual settings via MQTT and restore them after an event for notification reasons (phone calls and door bells) which results in several daily saving of a preset.

Other options to write the dara would be a great improvement.

Are there better controllers than ESP32 ?

Sep 29 '22 07:09 DarthWeber

Why would you update presets often (and via MQTT)?

What do you consider "better"?

Sep 29 '22 08:09 blazoncek

I am also on an ESP32 and I have had it happen 'randomly' to me once. However I can usually make it happen by saving a preset.

The first time it happened, it seemed like it added a NULL character immediately after the first preset. I haven't been able to get a NULL again, however I have gotten strange characters added in different locations (I will try to start tracking them).

Additionally, I am not sure if this is related, but when adding a new preset, it is re-using existing keys, which will make the JSON unreadable. If I use /edit and change the new key to a unique number and remove any extra added characters, then it loads fine.

Sep 29 '22 15:09 mobilla

Why would you update presets often (and via MQTT)? as mentioned above: For notifications of phone calls: Save currently effect as preset 99, play notification, restore saved preset 99

What do you consider "better"? "better" in this case means without these artefacts :-)

Sep 30 '22 04:09 DarthWeber

as mentioned above: For notifications of phone calls: Save currently effect as preset 99, play notification, restore saved preset 99

A waste. You can query which preset/effect is running by parsing response to /json/state and resubmitting the state after your notification is done. No need to store preset on the WLED device. Be mindful that too many writes to flash memory will wear it down, making it useless.

"better" in this case means without these artefacts :-)

Then ESP8266 is better. I am not sure everyone will agree overall though.

Sep 30 '22 13:09 blazoncek

This issue seems to have degraded to the point where presets are instantly corrupted as soon as a new preset gets saved or existing preset gets modified. On my ESP32 based Quinled DigiUno I cannot create any new presets or modify existing ones.

Do I need to throw the Quinled-esp32 in the trash and put a wemos d1 mini in it's place in order to have a stable experience again?

Oct 06 '22 23:10 gordonthree

Also seeing this pretty bad. I can no longer create a playlist without corrupting my presets.json and hard locking WLED on 13.3. Running a DigQuad.

My current presets.json. As soon as try to add a Playlist containing 8,9,10,11 I get connection to light failed and WLED goes offline until I hard reboot.

Oct 07 '22 15:10 SK360

If possible try 0.14-b0. Available from @srg74 repository or if you compile yourself. I can provide generic ESP32 binary without any extra options.

WARNING it is beta software with all implications this brings (unstable, not feature complete, etc)

Oct 07 '22 16:10 blazoncek

If possible try 0.14-b0. Available from @srg74 repository or if you compile yourself. I can provide generic ESP32 binary without any extra options.

So far it seems better... Been running 0_14 on my digiuno for several hours now and haven't seen the corruption return. Will try making a playlist tomorrow, that really seemed to cause corruption in 13.3

Oct 08 '22 06:10 gordonthree

Mind you, saving preset is still done in network callback. Prone to corruption. Everything else has been rewritten.

Oct 08 '22 06:10 blazoncek

I spoke too soon, updated a preset this morning to tweak the speed, the instant I hit save the esp crashed. After rebooting it, I get "there was an issue loading your presets"

On Sat, Oct 8, 2022, 01:49 Blaž Kristan @.***> wrote:

Mind you, saving preset is still done in network callback. Prone to corruption. Everything else has been rewritten.

— Reply to this email directly, view it on GitHub https://github.com/Aircoookie/WLED/issues/2793#issuecomment-1272246145, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEAWMMYTJLN2JIEWRNHISY3WCEKPJANCNFSM6AAAAAAQS6G6NI . You are receiving this because you commented.Message ID: @.***>

Oct 08 '22 14:10 gordonthree

If anyone wants to check: https://github.com/blazoncek/WLED/tree/async-psave It may still have issues, so use it only on test units.

Oct 08 '22 16:10 blazoncek

Nobody? Really?

Oct 10 '22 14:10 blazoncek

Nobody? Really?

Initial impressions are positive. I've updated a few presets and changed a playlist and I haven't corrupted anything yet. Will continue to play with it.

Oct 10 '22 15:10 SK360

There may be other issues since I had to change logic a bit. So any feedback is welcome.

Oct 10 '22 15:10 blazoncek

Compiled and burned async-psave to my DigiUno with ESP32, and was able to edit and save several times, also rebooted several times after editing, and presets remain uncorrupted.

However, integration with Home Assistant appears to be completely broken. HA can't see the state any more, and when I click on a control I get the error message:

This entity is no longer being provided by the wled integration. If the
entity is no longer in use, delete it in settings.

Should I open a new issue for this somewhere?

On Mon, Oct 10, 2022 at 11:49 AM Blaž Kristan @.***> wrote:

There may be other issues since I had to change logic a bit. So any feedback is welcome.

— Reply to this email directly, view it on GitHub https://github.com/Aircoookie/WLED/issues/2793#issuecomment-1273514224, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEAWMM5X5YK72RVB6ZXTEWLWCQ3KHANCNFSM6AAAAAAQS6G6NI . You are receiving this because you commented.Message ID: @.***>

Oct 10 '22 17:10 gordonthree

However, integration with Home Assistant appears to be completely broken.

This is due to beta version 0.14-bl0. Change it to 0.14 (package.json & package-lock.json) and run npm run build to have updated web page. Then compile again.

You can complain to HA team as well as they intentionally do not allow beta versions.

Oct 10 '22 18:10 blazoncek

Interesting. My 0.14-bl0 seems to be fine so far with HA 2022.10.2. Haven't used it much as I'm afraid of the integration crashing WLED like it usually does for me but will see if my automations run tonight.

Oct 10 '22 18:10 SK360

I'll try deleting the integration and adding again, maybe something else changed when it was crashing so often.

On Mon, Oct 10, 2022, 13:38 Matt Simmons @.***> wrote:

Interesting. My 0.14-bl0 seems to be fine so far with HA 2022.10.2. Haven't used it much as I'm afraid of the integration crashing WLED like it usually does for me but will see if my automations run tonight.

— Reply to this email directly, view it on GitHub https://github.com/Aircoookie/WLED/issues/2793#issuecomment-1273682592, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEAWMMYGNJ5MKHNLO6RM7ULWCRPB5ANCNFSM6AAAAAAQS6G6NI . You are receiving this because you commented.Message ID: @.***>

Oct 10 '22 18:10 gordonthree

FYI there is still one occasion where write operation happens from network callback. It happens when custom editing preset (adding elements not available in UI). This cannot be avoided unfortunately as web request is parsed (JSON) prior to determining it needs to be saved or not. In such case it is written to disk using JSON serialization.

Oct 11 '22 09:10 blazoncek

Feedback, please.

Oct 13 '22 05:10 blazoncek

With all the effort I pour into WLED it would be prudent of you to comment (provide feedback) on the proposed solution. I am going to close the issue as completed as no feedback usually means issue solved or no longer relevant. Reopen if you feel otherwise.

Oct 16 '22 12:10 blazoncek

WLED WLED copied to clipboard

presets.cfg corrupted after a while

What happened?

To Reproduce Bug

Expected Behavior

Install Method

What version of WLED?

Which microcontroller/board are you seeing the problem on?

Relevant log/trace output

Anything else?

Code of Conduct

WLED
WLED copied to clipboard