awtrix3 icon indicating copy to clipboard operation
awtrix3 copied to clipboard

[BUG] Animated images after firmware update to V0.98 cause system crash

Open UrsKobler opened this issue 6 months ago • 30 comments

Bug report

Describe the bug

If apps display animated images in addition to text, the system may crash, depending on the number of animated images used.

Afterward, the display can only be restarted from the frozen state by manually resetting it.

Additional information

  • Devices involved:

    • Model: Ulanzi Awtrix Smart Pixel Clock 2882 (TC001)
    • awtrix3 version: [v0.98 ]
    • used interface: HTTP API
  • My System requirements:

    • Platform Operating system: Linux
    • Architecture: Arm64
    • CPUs: 4
    • Speed: 2400 MHz
    • Model: Cortex-A76
    • RAM: 7.8 GB
    • Node.js: v20.19.1
    • NPM: 10.8.2

To Reproduce

Steps to reproduce the behavior:

  1. Use the HTTP API interface of the Ulanzi Awtrix Smart Pixel Clock.

  2. Create App-1 with the text 'App-1' and the animated image ID 50065.

  3. Create App-2 with the text 'App-2' and the animated image ID 50556.

  4. Create App-3 with the text 'App-3' and the animated image ID 51008.

  5. Create App-4 with the text 'App-4' and the animated image ID 55417.

  6. Create App-5 with the text 'App-5' and the animated image ID 61524.

  7. Create App-6 with the text 'App-6' and the animated image ID 7789.

  8. Display the apps on the screen and wait approximately 48 hours until the screen freezes.

Expected behavior

The display remains frozen and can only be reset by manually restarting. When using 6 animated GIF files, the display freezes after 48 hours of operation!

Screenshots

Image

Logs

The firmware crashes so no log file can be written.

Additional context

I suspect the error lies in the program component responsible for displaying the animated images (GIF files). Presumably, a memory leak occurs every time the image is displayed, which consumes RAM and causes the program to crash.

All 6 apps have been running smoothly for 14 days, as long as only the text and no animated images are displayed!

UrsKobler avatar May 13 '25 09:05 UrsKobler

Hey Urs,

I’ve now tried to reproduce your issue on my test-Ulanzi. I created 6 apps using the icons you mentioned (even in the same order).

No problems at all. It’s been running for 24 hours now, so I think we’ve clearly passed the “approximately one hour” mark. I don’t think it’s a firmware issue.

What have you already tried to solve the problem?

nob0dy80 avatar May 23 '25 13:05 nob0dy80

Hey nob0dy80,

Thank you for running the test I described to reproduce the phenomenon.

As already described, the system usually freezes after 48 hours of operation.

The problem can probably be reproduced more quickly by setting the cadence for displaying the animated GIF files as high as possible.

However, since the error usually reproducibly occurred after 48 hours in my described test environment, I assumed that other users would also experience it after 48 hours at the latest!

Since I am certain that the error reproducibly occurs after at least 48 hours, I have only operated the display with pure text output, as the display plays an important role in my production system for monitoring the energy flows of my PV solar system.

I would be grateful if you could continue the test, as I am sure that after 48 hours the display will freeze.

UrsKobler avatar May 23 '25 13:05 UrsKobler

Hi Urs,

Unfortunately, you mentioned two different time spans in your error report, which you just corrected. So we're not talking about roughly an hour, but actually about 48 hours. It would’ve been nice if you had said you made a mistake in the original report.

But no worries—I've got a few of these devices. I'll let it run for another 1–2 days and we’ll see what happens.

Image

nob0dy80 avatar May 23 '25 15:05 nob0dy80

Hello nob0dy80,

Please excuse my mistake with the incorrect test duration. I only realized after your message that I had made a mistake with the test duration of 1 hour.

Since the values ​​displayed on my display depend partly on the outside temperature, wind speed, and whether it's raining, the duration always varied slightly.

But after 24 hours at the latest, I was forced to manually restart the display.

Image

UrsKobler avatar May 23 '25 15:05 UrsKobler

What you're doing with it can largely be inferred from the icons.

I really like your wall mount for it :-) And above all, it seems like you've somehow made the power supply invisible. Very nice. Definitely a benchmark :-)

As I said, I'll just let it run for now, but I have a strong suspicion that the issue isn't with the firmware itself.

nob0dy80 avatar May 23 '25 15:05 nob0dy80

I doubt that as well. Awtrix now has a user base in the five-digit range. Having 6 apps with icons is nothing unusual otherwise I would have lots of issues this kind. I myself have 15 apps running 24/7 without any issues. I suspect it must be something else. Maybe you're using other API commands too?

Blueforcer avatar May 23 '25 16:05 Blueforcer

@Blueforcer Well, nothing's impossible... I'm thinking it might be caused by a faulty GIF icon. So I'll just let it run and see what happens. Since @UrsKobler didn’t mention what he tried to fix it, I’d generally recommend reflashing AWTRIX. But let’s see what time reveals...

nob0dy80 avatar May 23 '25 16:05 nob0dy80

If the error only occurs on my device and I'm the only one experiencing this strange phenomenon, I can only imagine that the cause could be a corrupted GIF file that may have been damaged during upload to the display.

If you haven't been able to reproduce the error until tomorrow, I'll try deleting all GIF files on the display and then reloading them.

UrsKobler avatar May 23 '25 17:05 UrsKobler

If the gif was really corrupt, then AWTRIX would crash the first time and not after hours. I also recommend switching off all other automations and only running /customapps. Maybe there is still an old automation running that either sends an implausible command or the API function has a bug. you can also track the RAM consumption in the stats.

Blueforcer avatar May 23 '25 17:05 Blueforcer

Since there's a lot of speculation right now, it might be better to talk in a day or two—once the test is done

nob0dy80 avatar May 23 '25 18:05 nob0dy80

In the meantime, I deleted all GIF files and then reloaded them onto the display. I then ran the test with the animated GIF files overnight.

Image

Unfortunately, this morning I discovered that the display had frozen again.

The awtrix-light FreeRAM showed the following values:

Test start 05/23/2025-8:36 PM: 113'796 bytes

Test end 05/24/2025-8:37 AM: 119,176 bytes

If I'm the only user in the world experiencing this phenomenon, I have to assume that something is malfunctioning with the hardware.

I could imagine that a thermal effect in the charging electronics could lead to the described phenomenon under higher loads due to the GIF images displayed.

Since the animated images are purely cosmetic, I'm fine with the purely textual information on the display, so I won't be buying a new display at the moment as long as it's still usable this way.

UrsKobler avatar May 24 '25 07:05 UrsKobler

Did you tried my other recommendations? And are you using a unifi WiFi?

Blueforcer avatar May 24 '25 09:05 Blueforcer

Hi Blueforcer,

I use the WiFi Mesh System from the German manufacturer AVM. It contains the Fritzbox 5530 AON router and several AVM access points connected via LAN.

I've disabled the internal apps (temperature, clock, etc.) since the beginning, and I don't use any custom apps either.

UrsKobler avatar May 24 '25 10:05 UrsKobler

II meant if you use any other APIs besides customapps.

Blueforcer avatar May 24 '25 10:05 Blueforcer

Hi Blueforcer,

For a good two years, I've been using a Raspberry Pi 5 based on the Linux Debian derivative operating system to control my smart home. The iobroker serves as the basis, which interfaces with the display using the installable Awtrix-Light adapter, which uses the HTTP request-based API for the Awtrix firmware.

Since the beginning, I've been using the Awtrix-Light adapter for my application, initially with Awtrix firmware 0.97. After the developer of Awtrix-Light further developed the adapter and recommended the latest Awtrix firmware for his latest adapter development, I followed his recommendations and installed the latest firmware release 0.98.

Since I'm sure Matthias Kleine would have contacted you long ago if he had discovered the phenomenon I'd noticed, I was confused and wanted to ask the inventor of the ingenious Awtrix firmware directly for his opinion.

If the error symptom I described cannot be reproduced in @nob0dy80's long-term test, I have to assume that my display hardware probably has a thermal error in the charging electronics, which coincided with the firmware change from V0.97 to V0.98.

UrsKobler avatar May 25 '25 10:05 UrsKobler

Hello Urs,

unfortunately, I cannot reproduce your error. It has now been running continuously for 52 hours. If your device is permanently supplied with power, it’s unlikely to be a heat problem with the electronics, since most of the heat is generated during charging. Simply displaying (whether gif or just text) is, at least on my side, very low at 2.1 watts.

If you haven’t done so yet, I would recommend reflashing the device and then only placing the 6 gifs that you need for your display. Otherwise, I do agree with you that it’s probably a hardware fault.

nob0dy80 avatar May 25 '25 12:05 nob0dy80

Hello nob0dy80,

First of all, I would like to thank you for your support in determining the cause of the error with my display.

As a final step, I will gladly follow your recommendation and reflash firmware version 0.98.

I will be happy to report on the outcome of the final long-term test.

Best regards, Urs

UrsKobler avatar May 26 '25 04:05 UrsKobler

Unfortunately, I have exactly the problem described here with two Ulanzi's My displays freeze every day. Sometimes after restarting after only 5 minutes

I reported this here a year ago and am always amazed that so few people seem to be affected.

Since the problem has existed for a very long time and there is no improvement in sight, I will now do without animated GIFs and reprogram my display.

warp735 avatar Jun 03 '25 04:06 warp735

As long as the problem is not reproducible, but only occurs with 2-3 users out of over 10k, I unfortunately have to assume a hardware defect. I hope you understand that my hands are tied here. I can't fix an avoidable bug that doesn't exist.

Blueforcer avatar Jun 03 '25 06:06 Blueforcer

@nob0dy80 @Blueforcer

In the meantime, I've flashed the Ulanzi Display TC001 again with the latest firmware.

I then enabled all the GIF files mentioned for the long-term test.

Unfortunately, the display froze again after about 30 minutes!

Considering that out of 10,000 users, only @warp735 and @UrsKobler are experiencing the problem on their displays, I have to assume that it must be a defect in the display hardware in these few devices.

Nevertheless, I'd still be interested to know if this is a defect in a specific manufacturing batch? Or if @warp735 and I were simply unlucky when purchasing our Monday devices?

Funnily enough, @warp735 mentioned that he had observed exactly the same phenomenon with two displays.

As already mentioned, I can continue to use the display without the fancy animated images, without it constantly freezing.

I would like to thank everyone involved who took the time to work out a possible explanation for this unpleasant phenomenon.

UrsKobler avatar Jun 03 '25 07:06 UrsKobler

@warp735

I'd also be interested to know what your environment looks like when you use the display.

As already mentioned, I use the display in the iobroker environment via the awtrix-light adapter from developer Matthias Kleine, with the latest version 2.0.0. This adapter uses the HTTP API.

I would appreciate your feedback.

UrsKobler avatar Jun 03 '25 07:06 UrsKobler

@UrsKobler would you send me a backup of your complete awtrix? Either via the backup funktionality or (even better) a complete flash dump. Then I need a list of jsons which you're sending to awtrix. And I need all of them.

Then I'm maybe able to test it with a 1:1 copy of your awtrix.

Blueforcer avatar Jun 03 '25 07:06 Blueforcer

If I only had one device, I might also consider a defect. But with two devices, that's unlikely. What's more, they work wonderfully with static images.

I also use the ioBroker adapter. But in the stable version 1.7.0

Incidentally, I have had this problem since all versions. So it's not specifically due to 0.98

warp735 avatar Jun 03 '25 07:06 warp735

@Blueforcer

In the meantime, I've created a backup of the entire awtrix using the backup function.

backup_awtrix_v089.zip

Apart from the animated GIF files mentioned above, I'm not sending any other JSON files to the display.

Image

Would you please explain to me how I can copy a complete flash dump from the display?

Thank you for your support.

UrsKobler avatar Jun 03 '25 10:06 UrsKobler

you can do it with esptools:

python esptool.py -b 115200 --port COM4 read_flash 0x000000 0x400000 AWTRIX_DUMP.bin

you need to send a json, otherwise awtrix cannot show any custom app. If youre using any third party integration you need to figure out what its sending to awtrix.

Blueforcer avatar Jun 03 '25 12:06 Blueforcer

@Blueforcer

In the meantime, I looked at the original manual for the Ulanzi TC001 Smart Pixel Clock. This is where I discovered for the first time that there is a hardware reset button on the back of the case.

Image

Image

After pressing the reset button, I had to log the display back into my SSID using the SSID password.

Since this process, the display has been running with all the GIF files I described for 12 hours.

To be sure that performing the hardware reset has resolved the issue I described, I will monitor the display for another 12 hours and report back on whether my problem has definitely disappeared.

UrsKobler avatar Jun 04 '25 04:06 UrsKobler

@Blueforcer

Unfortunately, I was too quick to celebrate the results of my first hardware reset.

I just noticed that the display has frozen again after about 17 hours.

So, this unpleasant phenomenon persists, and I have to, for better or for worse, get to grips with the esptool for creating the image dump, which is new to me. If I have any problems with it, I would be grateful if you could support me with your experiences.

UrsKobler avatar Jun 04 '25 05:06 UrsKobler

@UrsKobler Thats not a "hardware" reset button. It's a regular button wich is read by the ESP to do any kind of software stuff. In this case clearing the flash and settings etc which is also done during a reflash.

To be clear.. You're not using animated icons and its still freezing?

Blueforcer avatar Jun 04 '25 06:06 Blueforcer

@Blueforcer

The display uses animated GIF files and it still freezes 17 hours after pressing the ESP button! Without using animated icons, there is NO problem!

UrsKobler avatar Jun 04 '25 06:06 UrsKobler

@UrsKobler you may want to try out #564

eku avatar Jun 05 '25 05:06 eku