Screen displays corrupted images on Mac

[opless]

Someone else has had a similar issue here:

https://youtu.be/4z5OOW1jL94

I’m also using an M1 Mac.

I’m also reverse engineering the vendor .net binaries to attempt to figure out what is going wrong.

Though I’m struggling a little with the obfuscation. 😂

it’s not something as simple as endian issues!

[opless]

Looks like it's something weird with the serial comms or the operating system.

Closing because it's not fixable here.

[mathoudebine]

Sorry to hear that, FYI the display you should expect is this one: https://www.youtube.com/watch?v=Y1m4KK881Aw I had greater success in decompiling the chinese binary than the english one, it was not obfuscated. You will find the link of it in the README

[opless]

I've had a look at all three, and yes, the one you mention is MUCH easier to work with.

It's a shame there's no way to get hold of the creators and have a look at their firmware source.

I'm considering knocking something up in golang, to see if it's just the serial port drivers are broken on the M1 or if it's just pyserial that's not working.

[alanrenouf]

Im also getting this issue on my Mac, dont get it on windows. the port shows up as "/dev/tty.usbmodemUSB35INCHIPSV21" @opless did you find a workaround ?

[opless]

It's very much to do with the USB serial implementation, as far as I can tell.

I had some better luck sending data to it via golang, as the serial port initialisation is different.

I pretty much put the project aside after things getting busy with the day job and having less brain in the evening. It's certainly something weird with the M1 serial usb drivers though!

[mathoudebine]

Hi, i'm reopening this issue for people with Mac that have the same problem. Feel free to discuss here. What you can also try is changing the COM port settings here: https://github.com/mathoudebine/turing-smart-screen-python/blob/deb0a60b772f2c5acef377f13b959632ca649f9f/main.py#L184 If i remember correctly, the display also works with baudrates slower than 115200

[alanrenouf]

Tried slower baudrates and not getting any luck

[Ebag333]

https://github.com/mathoudebine/turing-smart-screen-python/pull/15

I think I've fixed it by serializing the updates to the screen (using a queue, so it'll run through each one cleanly before executing the next).

With the default code base I could get it to step on itself, once I started adding a bunch more things to update it just got worse and worse. You could help by increasing the delay after an update was done, but slowing down the updates isn't really a solution as it makes everything slow.

With this PR it runs at a very high rate and has no issues with keeping up, not getting serial errors anymore.

[alanrenouf]

I still get the corrupt image using the overhaul branch on my mac, I like the other changes to make the code more modular. Nice!

[gerph]

From my own investigation:

In my testing in my own library, I found that the delay between the bitmap data being output and the next command being issued needs to be quite long - about 0.02 seconds. Shorter than this and you get corruption on the display. The current delay in the code on the master branch is about 0.01 seconds, which means that the other slack is being taken up by the code sitting between the time.sleep(0.01) and the next SendReg writing a command. On faster machines that code will be executed faster, and result in the corruption (I believe).

In my library I changed the behaviour so that instead of explicitly sleeping at the end of the update of a bitmap, we remember when the last output happened and if we try to write a command before the required inter-bitmap-delay has expired, we sleep there instead. This means that we get useful work done between the two operations, and if you did enough work you'd never need to sleep at all before the next command was written.

My library changes that did this were here, if they're useful to you:

https://github.com/gerph/turing-smart-screen-python-mit/commit/8ba5901224ba5e52c37e86e668f73eca4d890b4d

It looks like the changes in PR #15, mentioned above, move the logic for the delay out of the communications library. I cannot tell how this keeps the bitmap transfers and the subsequent command updates apart enough to avoid the corruption.

[gerph]

I've tried out the feature/system-monitor branch, at f3748418c42, with flagship display, on my macOS system... I tried running the main a few times, and observed corruption on each of them - the first run generally doesn't even display the main image. The main image also appears offset for some reason.

https://user-images.githubusercontent.com/13813876/187958553-226441df-aeb4-486d-be07-a797d872448a.mp4

I'm not sure why it's doing that, but that's what I've seen with it so far - I may try and investigate later.

I suspect there's some race in the lcd_comm library - looking at lcd_comm_reb_b.py line 197, I see that we issue a SendCommand DISPLAY_BITMAP without the bypass_queue, and then go into loading the image and try to send it. Because bypass_queue wasn't set, the DISPLAY_BITMAP may not have been sent yet and the data for to send to the device may have been may get sent first. Additionally, the WriteData command isn't protected by the mutex, so it's possible that the queued data gets send DURING the image bitmap data.

[gerph]

Oh, I'm wrong about the data being sent during the bitmap update - SendLine is a queue operation, so it should be queued after the DISPLAY_BITMAP command. However, I think there's still a possibility that the SendCommand could be issued then something else get inbetween it and the data from the scheduler.

I'm really not at all convinced that this queue is the right way to go - for a start the queue code mixes the application-specific clients with the general queue dispatch, which means that if you want to re-use this code you have to copy out the queue handler code. And in any case, I'm not sure that it helps with any corruption problems at all.

[gerph]

I've tried replacing the queuing code with a simpler mutex-around-atomic operations with delay between the bitmap and next command. This has been solid on my flagship device. Unfortunately I cannot speak to the model A device... but I've tried making the necessary changes - it might not work at all on those models, and there might be a completely different processing model causing the problems there.

Here's the video of it working (again, tested 4 times to be sure it's not a fluke):

https://user-images.githubusercontent.com/13813876/188019709-de91bbd3-a9c6-45a7-8725-8c15cda36e8c.mp4

PR raised with the way that it worked for me. I've ordered a model A, but it'll be a while until it gets here.

https://github.com/mathoudebine/turing-smart-screen-python/pull/34

[gerph]

It still might be useful in some cases to have the serial comms on a separate thread, so it's possible the queue could be useful for that. However, that is still a bit complicated, and I didn't attempt it.

[9021007]

I have this issue as well, M1 Pro

[SaplingLTD]

Im facing same problem with the last version in MacOS. Model / Version A. Trying to figure out why it happened.

[gerph]

Oooh, it's built into a keyboard?! Is that a custom build or something that others can get hold of? (sorry I still don't know why there's corruption, though :-( )

[SaplingLTD]

So far, I can run the python code normally to connect and give command to the screen. I also tried to use this software on Windows https://wiki.keebmonkey.com/en/products/kbmscreen. It is recommended by the keyboard maker. And it worked fine. However, I want to use the keyboard with MacOS.

[mathoudebine]

Hi @SaplingLTD and welcome to this project

We didn't know the screen had been embedded into a keyboard! Looks really nice Can you try to use the branch fix/replace-queued-data-send-with-immediate-plus-deferred-delay made by @gerph instead of main branch? It may solve the issue with MacOS

It is actually in PR #60 because of issues when the program stops: sometimes after a relaunch the screen is in an invalid state and the program crashes. When relaunched again it will then work.

[amiltonjr]

@mathoudebine I just tested the branch fix/replace-queued-data-send-with-immediate-plus-deferred-delay in MacOS Ventura 13.1 with a Revision A display and it did not corrupt the display info! I think merging/rebasing it to the main will get it fixed and running with networking / GPU monitoring. @gerph if you do that, please let me know, then I'll test it out as well.

[phuongboss]

Confirmmed the branch fix/replace-queued-data-send-with-immediate-plus-deferred-delay fix the screen shuttering on macOS.

[itsWobble]

I tried using the branch #60 I'm getting a COMM error "00:06:31 [ERROR] Cannot find COM port automatically, please set it manually in config.yaml".

[nukdokplex]

Got this on Manjaro Linux

Here is logs:

мая 28 20:42:08 ndp-desktop python3[616]: 28.05.2023 20:42:08 [WARNING] Tray icon is not supported on your platform
мая 28 20:42:08 ndp-desktop python3[616]: 28.05.2023 20:42:08 [INFO] Display reset (COM port may change)...
мая 28 20:42:13 ndp-desktop python3[616]: 28.05.2023 20:42:13 [DEBUG] Auto detected COM port: /dev/ttyACM0
мая 28 20:42:13 ndp-desktop python3[616]: 28.05.2023 20:42:13 [INFO] HW revision A does not support backplate LED color setting
мая 28 20:42:13 ndp-desktop python3[616]: 28.05.2023 20:42:13 [DEBUG] Drawing Image: BACKGROUND
мая 28 20:42:13 ndp-desktop python3[616]: 28.05.2023 20:42:13 [DEBUG] Drawing Text: CPU_MODEL
мая 28 20:42:13 ndp-desktop python3[616]: 28.05.2023 20:42:13 [DEBUG] Drawing Text: GPU_MODEL
мая 28 20:42:13 ndp-desktop python3[616]: 28.05.2023 20:42:13 [DEBUG] Drawing Text: RAM_MODEL
мая 28 20:42:13 ndp-desktop python3[616]: 28.05.2023 20:42:13 [DEBUG] Drawing Text: DOWNLOAD_RATE
мая 28 20:42:13 ndp-desktop python3[616]: 28.05.2023 20:42:13 [DEBUG] Drawing Text: UPLOAD_RATE
мая 28 20:42:13 ndp-desktop python3[616]: 28.05.2023 20:42:13 [DEBUG] Drawing Text: DISK_USAGE
мая 28 20:42:13 ndp-desktop python3[616]: 28.05.2023 20:42:13 [INFO] Detected AMD GPU(s)
мая 28 20:50:44 ndp-desktop python3[616]: Exception in thread Queue_Handler:
мая 28 20:50:44 ndp-desktop python3[616]: Traceback (most recent call last):
мая 28 20:50:44 ndp-desktop python3[616]:   File "/usr/lib/python3.10/site-packages/serial/serialposix.py", line 621, in write
мая 28 20:50:44 ndp-desktop python3[616]:     n = os.write(self.fd, d)
мая 28 20:50:44 ndp-desktop python3[616]: OSError: [Errno 5] Ошибка ввода/вывода
мая 28 20:50:44 ndp-desktop python3[616]: During handling of the above exception, another exception occurred:
мая 28 20:50:44 ndp-desktop python3[616]: Traceback (most recent call last):
мая 28 20:50:44 ndp-desktop python3[616]:   File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
мая 28 20:50:44 ndp-desktop python3[616]:     self.run()
мая 28 20:50:44 ndp-desktop python3[616]:   File "/usr/lib/python3.10/threading.py", line 953, in run
мая 28 20:50:44 ndp-desktop python3[616]:     self._target(*self._args, **self._kwargs)
мая 28 20:50:44 ndp-desktop python3[616]:   File "/opt/turing/library/scheduler.py", line 74, in wrap
мая 28 20:50:44 ndp-desktop python3[616]:     scheduler.run()
мая 28 20:50:44 ndp-desktop python3[616]:   File "/usr/lib/python3.10/sched.py", line 151, in run
мая 28 20:50:44 ndp-desktop python3[616]:     action(*argument, **kwargs)
мая 28 20:50:44 ndp-desktop python3[616]:   File "/opt/turing/library/scheduler.py", line 64, in periodic
мая 28 20:50:44 ndp-desktop python3[616]:     action(*actionargs)
мая 28 20:50:44 ndp-desktop python3[616]:   File "/opt/turing/library/scheduler.py", line 163, in QueueHandler
мая 28 20:50:44 ndp-desktop python3[616]:     f(*args)
мая 28 20:50:44 ndp-desktop python3[616]:   File "/opt/turing/library/lcd/lcd_comm.py", line 99, in WriteData
мая 28 20:50:44 ndp-desktop python3[616]:     self.lcd_serial.write(bytes(byteBuffer))
мая 28 20:50:44 ndp-desktop python3[616]:   File "/usr/lib/python3.10/site-packages/serial/serialposix.py", line 655, in write
мая 28 20:50:44 ndp-desktop python3[616]:     raise SerialException('write failed: {}'.format(e))
мая 28 20:50:44 ndp-desktop python3[616]: serial.serialutil.SerialException: write failed: [Errno 5] Ошибка ввода/вывода

[evilpete]

I am seeing the issue, exactly like @SaplingLTD

MacAir (Intel Core i5) Monterey 12.6.8

[katanza12]

Same issue: MacBook Pro M1 Sonoma 14.0. I tried to remove background image, dynamic text and other graphics, put only a static text (starting with simple.program.py) but without success.

[a3rosol]

I'll pile on. M1 Ultra. I had to specify the com_port manually and it didn't work with the tty device but seems to work most of the time with the cu device. you can find them using ls /dev/{tty,cu}.* mine shows up as /dev/cu.usbmodemUSB35INCHIPSV21 . But no matter which theme I chose, I always get artifacts on the screen when refreshes happen.

[jeffsf]

Similar defects being seen on macOS 14.4.1 (Sonoma) and Intel in #512

If I've found the right object, changing from RTS/CTS to DSR/DTR did not seem to change the behavior

    # jmk -- try adjusting serial parameters
    lcd_serial: serial.serialposix.Serial = display.lcd.lcd_serial
    from pprint import pprint
    pprint(lcd_serial.get_settings())
    lcd_serial.rtscts = False
    lcd_serial.dsrdtr = True
    pprint(lcd_serial.get_settings())

    # Initialize the display
    display.initialize_display()

[Numbski]

Just chiming in - I am on an intel mac and getting this exact same behavior, so it isn't a CPU issue.

[Numbski]

Silly question - could it be the baud rate? Per the documentation, none of this is plain text. We're generating bitmaps and then pushing them to the serial port. Could be that the bitmap gets corrupted in the transfer process?

[opless]

It's something to do with the mac drivers, I'm sure.