MiniDexed

MiniDexed copied to clipboard

I open this separate issue, because we will probably have to deal with this problem for a while.

Problem description

When quickly playing on an USB MIDI keyboard controller, connected to a Raspberry Pi 1-3 or Zero (Raspberry Pi 4 is not affected), notes can get stuck, i.e. the respective tone is played forever, even after releasing the key. This can be reproduced with continuous voices (e.g. organ) playing rapid glissando for a while on the following keyboard:

• M-AUDIO Keystation Mini 32 Mk3

Workaround

The problem may be reduced or solved with the following option in the file cmdline.txt on the SD card:

usbspeed=full

This switches the whole USB from high speed (480 MBps) to full speed (12 MBps). The most USB MIDI controllers do only support full speed, so this is generally not a problem, but a few other keyboards do require high speed mode to work. The workaround does not work there and cannot applied by default.

There is a kernel image for the Raspberry Pi 2 attached, which is based on a quick hack. It does use the FIQ (Fast Interrupt Request) for the USB driver. It has a higher priority then the normal IRQ and using the FIQ may result in a more precise timing on the USB. The DWHCI USB controller of the Raspberry Pi 1-3 and Zero requires 8000 interrupts per second (one per microframe) for an accurate timing, which is not easy to fulfill.

@probonopd Can you please try this kernel7.img with the rapid glissando test on the RPi 2 with the M-AUDIO Keystation Mini 32 Mk3? Does the problem remain with this image? I would like to know, if it worth doing the (big) effort to generally support the FIQ for the USB in Circle.

This kernel image is fragile, because of the quick hack. I may even crash, then try again. Serial MIDI does not work with it, because it used the FIQ so far too, and there is only one FIQ supported on RPi 1-3 and Zero. Please test with and without the fast=true option in cmdline.txt. You will not see the bank select and voice change messages on the screen, but the LCD should work.

kernel7.zip

Thanks @rsta2, using kernel7.zip I can still get stuck notes with and without fast=true. I also get them by e.g., pressing 4 keys at the same time repeatedly for some time.

Cannot seem to reproduce this using my Raspberry Pi Pico tester.

Could it be something else` Maybe the M-AUDIO Keystation Mini 32 Mk3 is errorously sending the same key down message more than once (insufficient debouncing?) under some circumstances?

Can you think of a quick way to log all MIDI messages to a file or serial port?

On the Raspberry Pi 4 without usbspeed=full and without usbpowerdelay=1000 in cmdline.txt I get Canot set device address (19) when I attach the M-AUDIO Keystation Mini 32 Mk3 to either port, and I cannot play any notes. Using only usbpowerdelay=1000, I do indeed get no stuck keys on the Raspberry Pi 4.

@probonopd Thanks for testing! I did not expect this behavior. So using the FIQ will probably not help for the M-AUDIO Keystation Mini 32 Mk3. I have to think about a logging method and will come back with a solution.

So perhaps we have to add usbpowerdelay=1000 to cmdline.txt in any case. Some devices require a longer time until the power is stable. usbspeed=full does not have a function on the Raspberry Pi 4.

@probonopd With this patch, you can redirect all screen output to the serial interface at GPIO14 at 115200 Bps 8-N-1. The screen is not initialized with this. Serial MIDI does not work, of course.

serial-monitor.zip

I'm working on the FIQ support for the USB driver of Raspberry Pi 1-3 and Zero. I guess there is still room for timing improvement. But this will take longer and requires many tests.

@probonopd For now I would like to know, if there is something I could do for MiniDexed (e.g. support multiple Dexed instances, or "EDIT VOICE" menu in the LCD/Rotary Encoder UI)?

@rsta2 multiple Dexed instances would be tremendous. Eventually this could lead to support for TX816/TX802 style Performance sysex.

Voice editing via the LCD/Rotary Encoder UI would also be nice, but I think we should focus on multiple instances first.

@probonopd OK, I will work on supporting multiple Dexed instances now. Yes, Performance sysex should be loadable in the end, but for now I will use an .ini file to configure the different instances.

Here is one more way that I can get stuck notes:

• RPi 2, HDMI
• Keystation Mini 32 MK3
• usbspeed=full in cmdline.txt
• Set up 5 TG ** TG 1: normal ** TG 2: OMNI, detune 7, stereo shift halfway to R ** TG 3: OMNI, detune -7, stereo shift halfway to L ** TG 4: OMNI, detune 15, stereo shift all the way to R ** TG 5: OMNI, detune -15, stereo shift all the way to L
• Play 6 notes and keep holding them while playing around with Pitch Bend key/wheel
• Repeat a couple of times. Sometimes notes get stuck No fast playing is needed but the Pitch Bend are arriving relatively fast from the MIDI controller.

Note to self: Try to reproduce this issue with something else than the Keystation Mini 32 MK3 (e.g., A non-USB MIDI keyboard plus a USB-Midi converter)

I think we should not concentrate on the RPi 2 too much. Unfortunately I guess, it's too slow, at least without fast=true. I'm still trying to find a concept for improving the USB driver with using the FIQ. My first approach seemed to work first, but now I was running into problems. It will take longer. I know, that the problem is there, but I do not have a solution at the moment.

Thanks, no hurries. I just wanted to provide this additional information in case it is helpful for isolating the root cause.

@probonopd I finished my tests with the modified USB driver, which now can use the FIQ (Fast Interrupt Request) to handle interrupts from the USB, especially the SOF (Start-of-Frame) interrupts, which occur 8000 times per second. The FIQ has a higher priority than the IRQ, which was used before, and can interrupt even running IRQ handlers. I suppose the timing of the SOF interrupts is very precise now.

If this has been the problem before, it may work now with your keyboard controller. It would be great, if you would make another test with the attached kernel image for the Raspberry Pi 3. Please add fast=true to your cmdline.txt and remove usbspeed=full. If this works, please try without a cmdline.txt. Thanks in advance!

kernel8.zip

Thanks @rsta2. I can still get to stuck notes when playing 6 keys at the same time repeatedly on the Raspberry Pi 3 with fast=true for a while (while having 4 TGs active using the default performance).

Interestingly I cannot reproduce this issue when using a serial MIDI keyboard attached to a self-made USB MIDI adapter using an Arduino. So it seems that the issue is only with some USB MIDI devices. It would be interesting to know whether my M-Audio Keystation Mini 32 Mk3 is the lonely exception in this regard, or whether some other USB MIDI devices suffer from the same.

If you have any idea how I could test/compare the behavior of the M-Audio Keystation Mini 32 Mk3, please let me know.

@probonopd Thanks for testing! I don't think your keyboard is the only one, which shows this problem with this USB driver, but I guess much more will work, than will not work. Unfortunately the only method to find out, if some keyboard works, is to take it and try it. The USB MIDI support in Circle is available for five years now, and I heard about problems with only three different keyboards (including yours) in this time.

@rsta2 thanks for the hard work you keep putting into this, I really appreciate it. Don't feel obliged to spend excessive amounts of time on this keyboard just because I happen to have it. On the other hand, if you think it helps development, I'd be happy to lend it to you, postage paid.

@probonopd Thanks for your offer! Unfortunately I have currently no idea, what I could change on the USB driver to improve compatibility. If this changes, I will come back on it.

New hardware to play with: Using a Nektar Impact LX61+ MIDI controller I also have to use usbspeed=full or else I get very many stuck keys on a RPi 2.

Have you tested this keyboard on the RPi 3 with the USB FIQ test kernel above?

Yes, unfortunately the stuck notes still occured.

Using a Nektar Impact LX61+ MIDI controller even with usbspeed=full I get stuck keys (very, very occasionally) on a RPi 2 when running 4 TGs plus effects.

I have a cheap used M-AUDIO Keystation Mini 32 Mk3 here now. I can reproduce the problem. It seems with using the FIQ for the USB driver, the 0x202 errors do not happen any more, but the 0x402 errors (data toggle error) still happen. I'm trying to found out, why? Connected to a Raspberry Pi 4B, there weren't any problems, as expected.

Unfortunately it's not limited to cheap ones. So my hope is if it can be fixed for the Mini 32 Mk3 then it will also be fixed for others, like the Nektar Impact LX61+.

This is also my hope. But first the 0x402 errors must be fixed, which is difficult enough. :(

The problem occurs on a Raspberry Pi 3B with Raspberry Pi OS too. I can demonstrate this with the M-AUDIO Keystation Mini 32 Mk3. When pressing and releasing the MOD button, the keyboard sends B0 01 01 B0 01 02 B0 01 03 ... B0 01 7F B0 01 7E B0 01 7D ... B0 01 01 B0 01 00 B0 01 01 B0 01 02 and so on. When doing this for a while, one B0 01 NN is missing in the row of MOD messages. I also compiled the Raspberry Pi OS on my own and added a printk() message, which is visible in /var/log/messages, when a data toggle error occurs. This happens, when the expected MOD message is missing.

Also see:

https://github.com/NXPmicro/mfgtools/issues/76#issuecomment-453966001

Wow. So we are tripping over a bug in the Raspberry Pi hardware/firmware?

Seems so. As I understand the USB 2.0 spec. the problem is not the data toggle error itself. These errors can happen for example with electronic interference, but it should be possible to ignore them without losing data. Currently I do not know, how this is possible on the RPi 1-3.

Looks like you have spent way too much time on this nasty issue already. Especially if this can be reproduced on Raspberry Pi OS: Do you think we should contact someone from the Raspberry Pi team?

I think I'm not ready with this yet. At least the usbspeed=full workaround should work. If I remember well, it didn't do this for you. I still need some time to check this. We could open an issue in the RPi "firmware" or "linux" repo, but the problem should be 100% proved, before doing so. I have tested this with my own-built kernel so far only. Still need some time.

See: https://github.com/raspberrypi/linux/issues/4986#issue-1199763175

There is a solution for this problem for Raspberry Pi OS. I have to see, if I can implement this for Circle.

Even with maximum polling speed, I get lost MIDI events with Circle (every 2 minutes or so). This seems to be very critical with these (low-buffered) devices. What we can do is, calling Dexed::notesOff() or Dexed::panic(), when such condition is detected. I also have to add another option for cmdline.txt (e.g. usbboost=true). That's another workaround, but I cannot generally implement this behavior, because it is not appropriate for other applications.

And, because the FIQ is needed for the USB in this configuration, the serial MIDI interface will be less prioritized on the RPi 1-3. We have to test, if serial MIDI is quick enough with the IRQ then, or if we get lost MIDI events via the UART then too.