board: Keyboard FeatherWing on nrf52-feather

[btashton]

Summary

This adds support for using the Solder Party FeatherWing to the NRF52 Feather Board https://www.solder.party/docs/keyboard-featherwing/

Supported Hardware Interfaces:

• [x] LCD
• [x] Touch Screen
• [x] Keyboard (including 5-way and soft buttons)
• [x] NeoPixel
• [x] SD Card with card detect
• [x] Qwiic / STEMMA QT I2C port (can use the i2ctool to see attached devices)

Testing

LCD running NX demo application:

[patacongo]

Cool! what kind of performance to you get with the LCD? I imagine it needs a really high SPI rate and maybe DMA to be fully satisfactory.

NX is a windowing system. NxWidgets is the companion library for buttons and the like. LVGL is a non-windowed option. LVGL should have lower resource usage and probably better performance since less clipping involved.

Another option option is pdcurses. Ken Pettit did some great displays (and even added logic to remote the curses display. You would have to use a NxTerm for pdcurses.

[btashton]

Cool! what kind of performance to you get with the LCD? I imagine it needs a really high SPI rate and maybe DMA to be fully satisfactory.

So unfortunately this chip nRF52832 maxes out at 8MHz but the LCD can go much higher. There is another version the nRF52840 which is capable of 32MHz.

When I allocated a couple rows and wrote them via DMA it was ok for slow tasks like the background.

NX is a windowing system. NxWidgets is the companion library for buttons and the like. LVGL is a non-windowed option. LVGL should have lower resource usage and probably better performance since less clipping involved.

I agree, this was just easier to get going and I had never used NXGLIB before os I thought I would give it a try. One thing I noticed is it seems that each pixel was being written individually which is super slow compared to writing a "rectangle" which could be DMA. I did not dig into this yet.

Another option option is pdcurses. Ken Pettit did some great displays (and even added logic to remote the curses display. You would have to use a NxTerm for pdcurses.

Yeah that would be a cool little demo with the keyboard.

[patacongo]

One thing I noticed is it seems that each pixel was being written individually which is super slow compared to writing a "rectangle" which could be DMA. I did not dig into this yet.

No, the NX LCD interface outputs runs: A horizontal extent of pixels. This has been discussion that this limits performance for painting large regions that are not bounded by the line length. But this is not a performance limitation for drawing rectangles that are inherently bounded in the window.

The ILI9341 lower-half drivers probably does output SPI, one pixel at a time. That implementation is not part of the ILI9341 upper-half driver at drivers/lcd/ili9341.c. To output a line it does:

 655   lcd->select(lcd);
 659   ili9341_selectarea(lcd, col, row, col + npixels - 1, row);
 663   lcd->sendcmd(lcd, ILI9341_MEMORY_WRITE);
 667   lcd->sendgram(lcd, src, npixels);
 671   lcd->deselect(lcd);

So from the standpoint of the ILI9341 driver it outputs the entire run of pixels in one command (sendgram). Whether or not the lower half driver supports DMA or not is up to the board implementation of the sendgram() method of the struct ili8431_lcd_s interface.

[protobits]

One thing I noticed is it seems that each pixel was being written individually which is super slow compared to writing a "rectangle" which could be DMA. I did not dig into this yet.

No, the NX LCD interface outputs runs: A horizontal extent of pixels. This has been discussion that this limits performance for painting large regions that are not bounded by the line length. But this is not a performance limitation for drawing rectangles that are inherently bounded in the window.

I understand you are using nxgraphics with LCD interface right? As I mentioned in the past this really limited performance when wanting to transfer various lines in one go. I have in my TODO list to add support for putrun() to accept npixels > rowsize for such displays but I never got to do it. Anyway, as Greg says, this will not improve NX since putrun() is used once per line obviously. I used LVGL via a character driver which exposed the LCD interface and this allowed me to send putrun() for many rows. Contributing this to mainline is also in my backlog since I have to go back to test with another board to do so. If you're interested in trying it, it is here: https://gitlab.com/nuttx_projects/lcd_dev Note that it uses a non-standard putrows() since this was my solution to the aforementioned problem, but as we discussed previously, extending putrun() would be easier.

[btashton]

Yeah that is where I was saying having the ability for NX to render "rectangles" that are short in the X axis in one transfer would be nice because right now a vertical line is expensive.

I did set my gram byte swap buffer to the max of 127 grams for DMA and it improved the performance quite a bit. You can see it here: https://media.giphy.com/media/j736EdTwUBwQHiPKhS/giphy.gif

I am going to leave the performance bits aside for now and work on adding the other hardware support.

[patacongo]

Some of this is a little confusing to me.

First, let me recommend https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=139629398 which is my most recent attempt to describe the graphics sub-system.

One thing I noticed is it seems that each pixel was being written individually which is super slow compared to writing a "rectangle" which could be DMA. I did not dig into this yet.

No, the NX LCD interface outputs runs: A horizontal extent of pixels. This has been discussion that this limits performance for painting large regions that are not bounded by the line length. But this is not a performance limitation for drawing rectangles that are inherently bounded in the window.

I understand you are using nxgraphics with LCD interface right? As I mentioned in the past this really limited performance when wanting to transfer various lines in one go. I have in my TODO list to add support for putrun() to accept npixels > rowsize for such displays but I never got to do it. Anyway, as Greg says, this will not improve NX since putrun() is used once per line obviously. I used LVGL via a character driver which exposed the LCD interface and this allowed me to send putrun() for many rows. Contributing this to mainline is also in my backlog since I have to go back to test with another board to do so. If you're interested in trying it, it is here: https://gitlab.com/nuttx_projects/lcd_dev Note that it uses a non-standard putrows() since this was my solution to the aforementioned problem, but as we discussed previously, extending putrun() would be easier.

NOTE that LVGL does not use the NxGraphics sub-system. It uses a framebuffer driver that bypasses the entire graphics sub-system and interfaces directly to the LCD driver. No discussion regarding putrun in that context would apply in the NxGraphics context.

The framebuffer driver is a small bag hanging off the side of the graphics sub-system and not really integrated into the design.

I think it would be worthwhile to consider the overall graphics design and flow of data before focusing on some minute details that are not part of the graphics subsystem. I think we need to take a fully integrated approach to anything we do.

[protobits]

LVGL as it is now on NuttX indeed uses this "frambuffer character driver to LCD interface" but I really dislike this, it is an unnecessary confusion of interfaces. The character driver I wrote is a very simple mapping of LCD interface to userspace without performance penalty and clear semantics. LVGL can use whatever you want. See how I implemented the interface myself: https://gitlab.com/bicycle-companion/firmware/-/blob/master/extra_apps/bicycle_companion/display.cxx#L93

Yes, what I'm talking about does not impact NX in anyway. I was actually suggesting Brennan that he could try LVGL for this if he wanted, and how it could be made faster with this interface than with the framebuffer -> LCD interface. As per improving NX, I'm personally more vested in LVGL for my use cases so I cannot comment.

[btashton]

@patacongo Here is what the it looks like when NXGraphics writes

What we see on the left is the setup of the region it is going to write which is a single row at 0x00c5. The first bit is spaced out because it sends a command byte and then has to toggle the DC line and send parameter bytes. I could improve the ili9341 driver to send the parameters in one transfer which would improve things quite a bit in that section.

Then there is a gap where I'm guessing it is byte swapping the buffer.

Then we see two DMA transfers of 0xdcdc grams. This is actually one transfer but it has to be split because of the nRF SPI driver which only allows 0xff bytes to be sent at a time unless you use the DMA list feature which I dont quite understand. We are still limited by the size of the swap buffer anyway.

Then we see the start of the next row where we send the same selection region for writting just one row down. This row did not need a new setup to happen if we had just extended the region in the previous setup.

Now I understand there are complexities here. If we are going to clear the background we are not going to allocate a whole buffer for the entire display, but if I was going to draw a thin rectangle from the top to the bottom of the screen this would 240 full row transactions just to send 480 bytes of data.

Here is an example of drawing a skinny rectangle:

[patacongo]

I did set my gram byte swap buffer to the max of 127 grams for DMA and it improved the performance quite a bit. You can see it here: https://media.giphy.com/media/j736EdTwUBwQHiPKhS/giphy.gif

Yes, the performance looks great to me.

I cannot tell for sure from the video, but there should be text in the two little windows. Is it there? Or is there a text problem.

It would be cool to see something like the NxWM window manager running on that platform. It looks like you have the band with to do that.

[btashton]

Looks like the stmpe811 driver currently does not support SPI, so I will have to add that in. Also unfortunately the interrupt pin for this chip is not connected to the feather interface, but there is a very large test point that could be jumpered to support this. I'll try to get a PR up against this driver sometime today.

[btashton]

Ok most of this work ended up being spread over a few different PRs to add the driver support to NuttX. At this point I would like to get these board configurations merged in. I am still interested in trying to break this out a little bit later to make it easier to support this featherwing breakout board in other boards, but I would like to do that in another PR. I am somewhat motivated to do that because the nrf5232 was actually not a great choice for this board, the SPI hardware is very limited at 255 byte transfers and only 8MHz SPI clock. I know these SPI displays are generally not great, but this makes it a lot worse.

You can see an example of this running here: https://github.com/apache/incubator-nuttx-apps/pull/591