ev3dev
High CPU utilization with gyro in S1/S2 ports on EV3 on ev3dev-stretch
Describe the bug
Sometimes, one of the kworker theards ~gets stuck~ and continuously uses around 20 % of the EV3 CPU. EDIT: this is normal behaviour, nothing gets "stuck" in the typical sense. This seems to have effect on software microbenchmarks running on the brick.
~Around the same time, the following messages started appearing in dmesg:~ (this seems to be unrelated)
[ 3765.754471] lego-port port1: Removed device 'ev3-ports:in2:ev3-uart-host'
[ 3766.107344] lego-sensor sensor1: Unregistered 'lego-ev3-gyro' on 'ev3-ports:in2'.
[ 3766.128986] systemd[1]: Stopping LEGO MINDSTORMS EV3 UART Sensors...
[ 3766.210525] systemd[1]: Stopped LEGO MINDSTORMS EV3 UART Sensors.
[ 3767.328012] lego-port port1: Added new device 'ev3-ports:in2:ev3-uart-host'
[ 3768.049985] systemd[1]: Starting LEGO MINDSTORMS EV3 UART Sensors...
[ 3768.105481] serial8250: too much work for irq25
[ 3768.122672] systemd[1]: Started LEGO MINDSTORMS EV3 UART Sensors.
[ 3768.320323] serial8250: too much work for irq25
[ 3768.550315] serial8250: too much work for irq25
[ 3768.780253] serial8250: too much work for irq25
[ 3769.010184] serial8250: too much work for irq25
[ 3769.240212] serial8250: too much work for irq25
[ 3769.470190] serial8250: too much work for irq25
(...)
At the time, I had two large EV3 motors plugged into ports B+C and a single EV3 gyro to port S2 (model 49N3, likely in angle mode). The issue disappears after the sensor is unplugged and plugged back. If I remember correctly, the issue started earlier with another unplug/replug cycle,
To Reproduce ~I am not sure, this doesn't happen very often.~ Repeated plugging and unplugging of the sensor could lead into a system state where this happens. It could be related to high CPU/RAM utilization from other user programs too.
EDIT: plug in a gyro sensor to S1/S2 while turning it at the same time. After ev3dev detects the sensor, stop turning it. Kworker should be using around 20% of the CPU.
Expected behavior ~No kernel threads get stuck this way~. UART RX has lesser impact on the system
Robot (please complete the following information):
- Device: EV3 (not overclocked)
- Connection: Ethernet adapter (Belkin F5D5050)
- Versions: [see ev3dev-sysinfo]
System info (from ev3dev-sysinfo)
Image file: ev3dev-stretch-ev3-generic-2020-04-10
Kernel version: 4.14.117-ev3dev-2.3.5-ev3
Brickman: 0.10.3
BogoMIPS: 148.88
Bluetooth:
Board: board0
BOARD_INFO_HW_REV=7
BOARD_INFO_MODEL=LEGO MINDSTORMS EV3
BOARD_INFO_ROM_REV=6
BOARD_INFO_SERIAL_NUM=00165344CF21
BOARD_INFO_TYPE=main
Additional information
I had an ethernet adapter plugged into the USB host port, but I hope this is not what is causing the issue. The USB device port was empty.
Bus 001 Device 002: ID 050d:0121 Belkin Components F5D5050 100Mbps Ethernet
I also intentionally stopped many of the system services to free up some RAM. While this could be related (stopping bluetoothd could mess with the link to the bluetooth chip), I'd also hope this doesn't cause this issue.
The too much work for irqX for UARTs on input ports has been a long standing issue - although I'm surprised I can't find an existing issue dedicated to it.
The EV3 UARTs have 16 byte hardware buffers and if the CPU is too busy to service the UART interrupts, then we can end up with a hardware buffer overrun. This happens when sensors are sending large and/or frequent data.
I was able to reproduce the CPU usage issue by plugging an EV3 Gyro sensor into port 1 or 2. Interestingly, I don't see the same CPU usage when an EV3 Gyro sensor is plugged into port 3 or 4. These port use the PRU to emulate UARTs as opposed to the hardware UARTs used on ports 1 and 2.
Thank you for your reply, this does seem to fit the description.
I was able to reproduce the CPU usage issue by plugging an EV3 Gyro sensor into port 1 or 2. Interestingly, I don't see the same CPU usage when an EV3 Gyro sensor is plugged into port 3 or 4. These port use the PRU to emulate UARTs as opposed to the hardware UARTs used on ports 1 and 2.
I can partially confirm this - gyro in S3 was working OK. However, I tried this only once, so my observation doesn't have much weight.
I have some new observations. The serial8250: too much work for irq25 message seems to be unrelated. I am now getting high CPU usage even without that message (I saw it earlier, but not the last time I plugged it in). However, increasing the DMA interrupt triggering threshold seems to lessen the impact:
echo 14 | sudo tee ./devices/platform/soc@1c00000/serial8250.0/tty/ttyS0/rx_trig_bytes
After this, CPU usage from that kworker process goes down to ~7.6 %. Switching it back to 1 byte puts the usage back in the 18-20 % range.
Oh, I see why it happens only sometimes... EV3 gyro gets calibrated when connected. Sometimes I was lucky and the calibration was precise enough that the sensor was not sending the updated angle very often (it didn't change from its perspective). Therefore, the kernel didn't have to receive as much data.
With PRU SUARTs, I see slightly elevated CPU usage as well, but it is only around 2-3 %.
Hmmmm, the LMS2012 stock firmware was driving the UART directly and it also had a dedicated interrupt handler for each one (Uart1Interrupt/Uart2Interrupt). I'm wondering if the serial8250 driver isn't behaving suboptimally somehow.
On the other hand, stock firmware wasn't using DMA & the interrupt was triggered for each byte too.
EDIT: The same elevated CPU usage could be happening even on the stock firmware though, I haven't done any comparisons and even if I did, I don't know if the old kernel accurately tracks time spent in interrupt handlers.
I only now noticed https://github.com/ev3dev/ev3-kernel/commit/0ddd8d1710ae67ed5f1b226046e4defafcafbd11#diff-665093a901ce66d9b3ca5f30c135f53b19038fa0c68628396f793a35396fa2fe, so the high CPU usage a too much work for irqX issues should actually be "inversely" related.
Couldn't the other two UARTs also be managed from PRU1? It could at least move bytes to a larger queue that would then be handled by the main CPU. However, it also seems that PRU1 already handles I2C (I found some references to this in the EV3 DT), tacho counters and PWM, so it might be difficult to find free time slots for this.
I don't think the hardware UARTs currently use DMA. But if they could be configured to use DMA to read into a circular buffer, that would be better than using the PRU.
The information I found so far leads me to believe this should be possible. According to the AM1808 datasheet, the UARTs are connected to the EDMA controller somehow:
The thing that led me to think that the UARTs were already using DMA was that the UART controller is getting initialized with DMA enabled (UART_FCR_DMA_SELECT bit). However, reading from a port likely isn't how a DMA transfer is initiated :sweat_smile:
While not directly related, AM33xx UART driver seems to support DMA, so the concept was already realized. https://github.com/ev3dev/ev3-kernel/blob/8f4dfb355aae339ebff6543ed3edfaf0847a95c9/arch/arm/boot/dts/am33xx.dtsi#L344
Unfortunately, I do not currently have that much free time to dive deeper or to try to hack something together (I don't have experience with this yet).
On the other hand, it seems that AM335x and AM1808 have very similar UARTs, their register descriptions in datasheets match almost perfectly. Perhaps the 8250_omap driver could be extended for AM1808 (or DA830?) too.
Hmmm, I was wrong, I compared a PRU-ICSS UART in AM335x to the AM1808 one. The real AM335x UART has many more features.
I tried to self-reflect on whether I'm skill-wise up to the task and I think the answer is unfortunately no. I would like to help with this, but notwithstanding my interest in low-level stuff, I do not have the necessary experience and knowledge. I roughly know what DMA does and how approximately the UART peripheral on Arduinos works, but that is where I end. If I tried to write a driver, it would likely be a strange frankenstein composed of bits of other drivers.
On the other hand, the impact of this issue isn't big, sensors do work. Care needs to be taken when measuring performance of programs to have comparable results, but that might be it.
Please let me know if I can be of any help in the future.