LibVNC
MCU/FreeRTOS support
Is your feature request related to a problem? Please describe. No
Describe the solution you'd like I have been working with micro controllers that have attached touch screen displays and writing a binding for a GUI framework to MicroPython. One of the things I am striving to provide to the users is fast prototyping. Currently a user is able to write Python code and it will run on a micro controller for creating a GUI to be shown on an attached display. The binding is also able to be compiled to run on macOS and also Linux and this allows a user to be able to develop without needing to use an MCU with an attached display. While the ability to test a GUI on a PC does make things easier it has a downside, no hardware related bits are available. So NO GPIO access and things like that. This would make sense seeing as how the binding is running on a PC and not on the actual MCU.
What I am wanting to do is allow the user to be able to develop without the need to have a display attached to the MCU. Using the RFB protocol would be the best solution because just about every OS available has some form of VNC client software available. Ideally I would like to make thing easier by using an already existing library. The only hangup I am having is all the libraries I have come across handle to actual receiving and transmitting of the data. This is something that I would not use because the MCU I am using doesn't have the same API for networking as Linux et al. has. The library would be used to handle the reading of the incoming data and formatting the outgoing data but the actual sending and receiving would be done using code that I would write. This could be done easily if there are functions that would handle the reading of the data and creating the packets needed for sending.
The MCU I am working with uses an Xtensa 32-bit LX7 processor with 2 cores @ 240mhz. Depending on how it's configured it can have 2mb-8mb of RAM and 4mb-32mb of flash based program storage.
The first thing to hurdle is getting the library to compile using a gcc like compiler that is made for these specific MCU's. I don't know enough about CMAKE to know if adding it to my projects CMAKE build will carry over using a specific compiler and also how to set option flags that would normally be done by passing them in the build command.
The processor that I am using is dual core and I have the ability to set specific code to run on specific cores. I see that this library supports pthread but unfortunately this is not available. FreeRTOS is what is used to handle "threads". while the functionality of how the threading works is pretty close to pthread the API is completely different. I did come across a library that is a wrapper around FreeRTOS to provide a pthread API. I have not tested this to see if it will work. I am pretty sure there is going to be things that I will need to modify to get it to run on the MCU I am using. If there were macros that could be changed by the user to point to a different function for thing like creating a mutex and creating a thread this would make you library very flexible.
Describe alternatives you've considered I have looked at quite a few different VNC server libraries as a possible solution. Your library is the most complete in terms of much of the RFB protocol API is supported and the way it is written also would make it easier to make alterations that would allow it to be run on a micro controller.
Additional context Now that I am thinking about it providing macros to handle using encryption libraries other than the ones directly supported by the library would be of added benefit as a lot of MCUs have hardware accelerators for doing the encrypting and decrypting.
You can build without threading support, the rest is probably reseaching cmake and FreeRTOS. If your efforts are successful, let me know; would be happy to merge in some documentation w.r.t. embedded systems!
That would really make the library truly cross platform. What I am wanting to do is bare metal compiling so no OS. Not really an embedded system.
question tho. are there functions that are public that can encode and decode the VNC packets without having to use any of the built in connection code. The network related API for the ESP32 doesn't follow POSIX standards, not 100% anyway. so I would only use the library to encode and decode the packets. the library would handle reading the data that I would feed into it via a function call and it would work like it would normally would when doing that. for sending messages I would only need the library to build the packets when transmitting.
I guess your best bet is to write/get-from-a-third-party shims for BSD socket functions; there might be other issues as well. But at least documenting those here in public would be nice.
I am still crossing the bridge to get it to compile. It appears the only way to set the config options is I have to add the project as a sub directory. That is the only way the environment will carry over to it.
There is no way to just use the packet encoding/decoding portions of the library? I could write wrapper code for the network functions to make it work.
I am actually pretty damned close to getting it to compile. There are a few issues however.
You have some macro definitions that are redefinition's. you don't have any guards in place to see if they are already defined. I am thinking you may want to do this...
you have INADDR_NONE being defined in both the server and the client. Not sure why but they are bumping heads. They also bump heads with it being defined in lwip (light weight IP stack) which is what is used by the esp32. Nice thing about lwip is it has posix compat headers YAY!! and between that and also the posix compat header that can be gotten for FreeRTOS that is what is allowing me to compile your library on the MCU. I did have to write a new cmake script to handle ti but that wasn't too big a deal to do.
I am going to manually add the guard for that macro and see if there is anything else that causes problems. at the moment that is all there appears to be.
If you got a simple PR with additional header guards, I'll happily merge this.
I got sidetracked on something else and I have not put a guard in for the one error I am getting. Once I have it compiling I will submit a PR for any changes to the code I have made,
OK so I am making headway. so far a simple include solves the issue. In include/rfb/rfbproto.h there is this code...
#ifndef INADDR_NONE
#define INADDR_NONE ((in_addr_t) 0xffffffff)
#endif
and it's awesome that there is a guard in place to check to see if it is already defined. Issue is this macro gets set before sys/socket.h is included anywhere in libvncserver. sys/socket.h gets included after the macro gets set and the issue is there is no guard in inet.h which is where the macro gets set a second time.
There are a couple of ways to handle this.
- include
sys/socket.hininclude/rfb/rfbproto.hprior to the macro getting set. - tell the user that they must include
sys/socket.hin their project prior to including any part of libvncserver
Personally I believe that option 1 is the best way to go about it because it is pretty much a gurantee never have anyone open an issue should they not read the docs or they missed the line where it informs them of option 2 above. It's not going to cause any kind of an issue to include sys/socket.h before that macro gets set as it is something that ends up being used in libvncserver anyhow.
I changed the lines seen below which are located in include/rfb/rfbproto.h
#if defined(WIN32)
typedef int8_t rfbBool;
#include <sys/timeb.h>
#include <winsock2.h>
#endif
to read
#if defined(WIN32)
typedef int8_t rfbBool;
#include <sys/timeb.h>
#include <winsock2.h>
#else
#include <sys/socket.h>
#endif
and that solved that issue.
Next issue is a bit more complex and it is one that is going to take me a bit to figure out how to solve. Perhaps you might be able to help me out with the best solution.
This is the issue.
In a multitude of places errno.h is included in the esp32 build. The esp32 SDK (ESP-IDF) has created replacement headers for some things so it aligns with the posix API. Not all things have been created which is why there is a need to use the compat headers from lwip and also from another library that is a wrapper around FreeRTOS to provide a posix API for creating threads. Now here is the issue.
Includes are handled so when an include is wrapped in double quotes the way the compiler does the search is local path from where the include is taking place and then it searches the paths that are provided using the -I compiler argument. When an include is wrapped in gt and le the compiler searches the paths provided by the -I compiler argument first and then it looks for includes at the system level.
Se here is the issue.... lwip provides an errno.h as does the FreeRTOS wrapper and the ESP32 SDK provides one as well. since the file exists in 3 locations the first one that the compiler comes across when searching the paths is going to be the one that it is going to use. The issue is not all of the needed definitions exist in a single one of those files. The only way I can think to handle this is to create my own errno.h file and include each of the other 3 files using the abolsute path to each file and wrapping each one in double quotes. Now I am able to collect the paths programmatically using cmake, no issue there. what I do not know how to do is to create the file using cmake. This is something that you might be able to assist me in doing. In order to do this I would be using part of the ESP-IDF build system to collect the paths to the files. I think it would be better if code was written that wouldn't be tied specifically to the ESP-IDF build system. If there is a way to search the include paths supplied to cmake to locate header files that are the same name and create a new header file if multiples are found and include the multiples in the new header file and set the path to the generated file so it will be what is used when included.. I think that is the proper way to go about it.
It would take me days to figure out how to do that with cmake so it would do a proper recursive search so the paths and filename match properly and create the same path and filename.
My 2 cents here: For the time being, don't bother to automate this with cmake, let humans do it. I think a section in the README describing the very special steps for the errno.h situation on esp32 would be enough.
OK so I am getting down to the wire with this...
couple of errors I am not sure how to fix.,...
/vnc_compile_test/lvgl_micropython/lib/libvncserver/src/libvncserver/sockets.c: In function 'rfbProcessNewConnection':
/vnc_compile_test/lvgl_micropython/lib/libvncserver/src/libvncserver/sockets.c:480:19: error: storage size of 'rlim' isn't known
480 | struct rlimit rlim;
| ^~~~
/vnc_compile_test/lvgl_micropython/lib/libvncserver/src/libvncserver/sockets.c:509:8: error: implicit declaration of function 'getrlimit' [-Werror=implicit-function-declaration]
509 | if(getrlimit(RLIMIT_NOFILE, &rlim) < 0)
| ^~~~~~~~~
/vnc_compile_test/lvgl_micropython/lib/libvncserver/src/libvncserver/sockets.c:509:18: error: 'RLIMIT_NOFILE' undeclared (first use in this function)
509 | if(getrlimit(RLIMIT_NOFILE, &rlim) < 0)
| ^~~~~~~~~~~~~
/vnc_compile_test/lvgl_micropython/lib/libvncserver/src/libvncserver/sockets.c:509:18: note: each undeclared identifier is reported only once for each function it appears in
/vnc_compile_test/lvgl_micropython/lib/libvncserver/src/libvncserver/sockets.c:480:19: warning: unused variable 'rlim' [-Wunused-variable]
480 | struct rlimit rlim;
| ^~~~
/vnc_compile_test/lvgl_micropython/lib/libvncserver/src/libvncclient/listen.c: In function 'listenForIncomingConnections':
/vnc_compile_test/lvgl_micropython/lib/libvncserver/src/libvncclient/listen.c:91:18: error: implicit declaration of function 'wait4'; did you mean 'wait'? [-Werror=implicit-function-declaration]
91 | while ((pid= wait4(-1, &status, WNOHANG, (struct rusage *)0))>0);
| ^~~~~
| wait
/vnc_compile_test/lvgl_micropython/lib/libvncserver/src/libvncserver/stats.c: In function 'messageNameClient2Server':
/vnc_compile_test/lvgl_micropython/lib/libvncserver/src/libvncserver/stats.c:83:42: error: format '%X' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Werror=format=]
83 | snprintf(buf, len, "cli2svr-0x%08X", type);
| ~~~^ ~~~~
| | |
| | uint32_t {aka long unsigned int}
| unsigned int
| %08lX
/vnc_compile_test/lvgl_micropython/lib/libvncserver/src/libvncserver/stats.c: In function 'encodingName':
/vnc_compile_test/lvgl_micropython/lib/libvncserver/src/libvncserver/stats.c:157:38: error: format '%X' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Werror=format=]
157 | snprintf(buf, len, "Enc(0x%08X)", type);
| ~~~^ ~~~~
| | |
| | uint32_t {aka long unsigned int}
| unsigned int
| %08lX
/vnc_compile_test/lvgl_micropython/lib/libvncserver/src/libvncserver/httpd.c: In function 'validateString':
/vnc_compile_test/lvgl_micropython/lib/libvncserver/src/libvncserver/httpd.c:661:22: error: array subscript has type 'char' [-Werror=char-subscripts]
661 | if (!isalnum(*ptr) && *ptr != '_' && *ptr != '.'
| ^~~~
/vnc_compile_test/lvgl_micropython/lib/libvncserver/src/libvncclient/vncviewer.c: In function 'rfbGetClient':
/vnc_compile_test/lvgl_micropython/lib/libvncserver/src/libvncclient/vncviewer.c:343:36: warning: cast between incompatible function types from 'rfbBool (*)(rfbClient *, int, int)' {aka 'signed char (*)(struct _rfbClient *, int, int)'} to 'void (*)(struct _rfbClient *, int, int)' [-Wcast-function-type]
343 | client->HandleKeyboardLedState = (HandleKeyboardLedStateProc)DummyPoint;
| ^
This is probably all stuff not in FreeRTOS. I recommend commenting out vigorously and getting stuff to build, take notes, then clean up later.
PS: Also, you probably don't need libvncclient, so errors there can be ignored.
There is a use case for having the client which is providing a client to the user. LVGL is a light weight graphics library so providing a client would be easy to do because all of the graphics bits exist.
There also seems to be no way to compile only the server without it compiling the client too. Can I omit the client source files from being compiled? There doesn't appear to be a macro available to turn off the client being compiled.
I don't know what wait4, rlimit and getrlimit are apart of. perhaps it is something I would be able to implement. Just need to know what they are. Is there an alternative way. Perhaps there is an alternate way of providing what those things are providing to lbvncserver???
can the http server be turned off? does it need to compile when it is not being used?... We only plan on using sockets to send and receive data so there really is no need to have the http code compiled. I don't know if there is a dependence on it being compiled.
There is a use case for having the client which is providing a client to the user. LVGL is a light weight graphics library so providing a client would be easy to do because all of the graphics bits exist.
There also seems to be no way to compile only the server without it compiling the client too. Can I omit the client source files from being compiled? There doesn't appear to be a macro available to turn off the client being compiled.
No, both always built as of now.
I don't know what
wait4,rlimitandgetrlimitare apart of. perhaps it is something I would be able to implement. Just need to know what they are. Is there an alternative way. Perhaps there is an alternate way of providing what those things are providing to lbvncserver???
No idea, this code is very likely from a time before I entered the project.
can the http server be turned off? does it need to compile when it is not being used?... We only plan on using sockets to send and receive data so there really is no need to have the http code compiled. I don't know if there is a dependence on it being compiled.
Not as of now, in principle of course yes. It's not essential at all.
Not as of now, in principle of course yes. It's not essential at all.
How do I go about turning it off? There is no macro that controls this. would I need to remove the source files from the list to be compiled?
I am thinking there should be some config macros added to the source files and header files that control what does and doesn't get compiled. This is easier than having to modify the build script to control turning on and off parts of the library that could be considered as optional.
IDK if the server code has any dependency on the code in the client if it doesn't then the client would be an optional thing. The same with the client being dependent on any of the code in the server side of things. I see there is a common folder and I would imagine anything that both the client and server both use would be located in there. That kind of indicates that the server code should be able to compile without the client code and vice versa.
ok so here is some information in wait4 It is in sys/wait.h and this is what the man pages say for it.
These functions are nonstandard; in new programs, the use of waitpid(2) or waitid(2) is preferable.
https://man7.org/linux/man-pages/man2/wait4.2.html
so a code update to use waitpid or waitid is going to be the best solution to correct that error.
fixing the issue with 'getrlimit` should be pretty easy to do as well. need to add a check to the build system to see if that function is available. Right now a check is done to see if the resource.h file exists instead of check to see if the specific function that is needed exists. pretty simple thing to correct.
Thank you for your contribution, your work has realized what I have always wanted to do but have not been able to do!
What encodings do you currently want to support? All or just some encodings (due to the performance of the MCU)?
what do you mean by encodings? are you referring to the color format? if that is the case then RGB565, RGB888, xRGB888 and ARGB888. we will use no encryption due to the additional overhead because of using software encryption. Possibly in the future add encryption but one what is supported at the hardware level on the MCU.
I already wrote the code that converts the keysyms that are not control/function keys to UTF-8 and also single byte ASCII is those codes are received.
being able to do file transfers will be a handy thing to have working. IDK if your library handles writing a pointer to the frame buffer data or not, if it does that could be used but I think that the GUI frameworks handling of rendering a cursor might be better.
Functionally how it is going to work is the RAW RGB framebuffer data is going to get handed off to the VCN server if there is a client connection. The user will be able to choose whether or not to blank the display when a client is connected to VNC or to leave it displaying what is seen on the VNC connection. Could even use it to go one step further and set up a kind of VM design where more than one user is able to access the thing and have 2 completely different UI's showing over the connection.
MicroControllers these days are pretty robust, some of them being 4 cores @ 1.4ghz and they have hardware graphics accelerators built in. The GUI framework is able to render to a 1280 x 1024 display having a framerate of 60fps on some MCU's. That's pretty quick. That's a very large amount of data to send to a display to able to maintain 60fps refresh rate on the display. it comes out to 176947200 bits a second. or 176mbit transfer speed to the display. The limitation is not going to be the MCU as far as processing ability so much as it is going to be a limit of the network connection like WiFI which is only going to be able to send data at 30mbit a second. I will hammer out some code that will decide what color format to use based on how fast the data is transferring.
I do have a question about how exactly the data is handled on the client.
I know that there are 2 ways that frame buffer data can be transmitted. It can be transmitted when the client requests it or the server can send it when it is told to send it. Now my question is does the entire frame buffer get sent when a request is made or when the server needs to send an update? or does only the updated parts get sent?
I am hoping that only the parts that have been updated are what gets sent. I mam thinking the client should be writing data to a frame buffer that it has made for the entire display and when an update is received that new data gets written to the frame buffer and then the frame buffer gets rendered on the client side. The protocol specification doesn't go into much detail about the handling of this and if partial frame buffer data is able to be sent.
Also, I am going to submit another PR and this PR is for enabling/disabling the server and client code. It stops the code from being compiled. program storage is rather skinny on an MCU so the smaller we can make the compiled binary the better. don't need to have the client code on the MCU if it is not going to be used.
I know that the changes I made are not going to work because the test suite is not divided by client and server. It would be easier if someone more familiar with the test suite could separate the client and server tests so we could have the CI build only the client, then only the server and then both... this way it would ensure that everything is working correctly.
what do you mean by encodings? are you referring to the color format? if that is the case then RGB565, RGB888, xRGB888 and ARGB888. we will use no encryption due to the additional overhead because of using software encryption. Possibly in the future add encryption but one what is supported at the hardware level on the MCU.
I already wrote the code that converts the keysyms that are not control/function keys to UTF-8 and also single byte ASCII is those codes are received.
being able to do file transfers will be a handy thing to have working. IDK if your library handles writing a pointer to the frame buffer data or not, if it does that could be used but I think that the GUI frameworks handling of rendering a cursor might be better.
Functionally how it is going to work is the RAW RGB framebuffer data is going to get handed off to the VCN server if there is a client connection. The user will be able to choose whether or not to blank the display when a client is connected to VNC or to leave it displaying what is seen on the VNC connection. Could even use it to go one step further and set up a kind of VM design where more than one user is able to access the thing and have 2 completely different UI's showing over the connection.
MicroControllers these days are pretty robust, some of them being 4 cores @ 1.4ghz and they have hardware graphics accelerators built in. The GUI framework is able to render to a 1280 x 1024 display having a framerate of 60fps on some MCU's. That's pretty quick. That's a very large amount of data to send to a display to able to maintain 60fps refresh rate on the display. it comes out to 176947200 bits a second. or 176mbit transfer speed to the display. The limitation is not going to be the MCU as far as processing ability so much as it is going to be a limit of the network connection like WiFI which is only going to be able to send data at 30mbit a second. I will hammer out some code that will decide what color format to use based on how fast the data is transferring.
Sorry for the confusion, I am referring to
https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#76encodings
Ahhhh, compression. I would have to run tests to see if the time it takes to do the compression would give better performance than transferring the raw data. It all depends on how small the data that gets sent is. we also have to deal with compatibility of the external compression libraries and the MCU. That will be something that I can mess with after getting uncompressed (raw) to work.
It compiles!!! WOOOOO... Both the server and the client compile. dunno if they actually work just yet but to cross compilation using the xtensa GCC compiler was a success.
Not as of now, in principle of course yes. It's not essential at all.
How do I go about turning it off? There is no macro that controls this. would I need to remove the source files from the list to be compiled?
As of now, yes. A CMake switch could be added, but is not there (yet).
I am thinking there should be some config macros added to the source files and header files that control what does and doesn't get compiled. This is easier than having to modify the build script to control turning on and off parts of the library that could be considered as optional.
Yes, am open for it.
IDK if the server code has any dependency on the code in the client if it doesn't then the client would be an optional thing. The same with the client being dependent on any of the code in the server side of things. I see there is a common folder and I would imagine anything that both the client and server both use would be located in there. That kind of indicates that the server code should be able to compile without the client code and vice versa.
You're correct. server does only depend on common, client depends on common. no client<->server dependency. The configure logic is simply not (yet) in CMake :-)