GUI support

[kisvegabor]

Hi,

I've just discovered this project and I really like that it can run on MCUs too.

I'm from LVGL and I wonder if you are interested in integrating a UI engine into wasm3.

[vshymanskyy]

Hi! Nice to meet you. I'm fond of LVGL, and I'm wondering if we could have a call / discussion of the vision, format and some technical details. Would it be convenient for you to join our Discord channel? https://discord.gg/qmZjgnd Thanks much!

[huming2207]

I also have some ideas about this. I think LVGL can fit in Wasm3 perfectly and some (but maybe not all) functions bindings can be generated by LVGL's official tool. See here: https://github.com/lvgl/lv_binding_micropython/tree/master/gen

Basically, we can either 1. modify this Python script, to make it generate Wasm3 binding instead of MicroPython binding, or 2. make use of the JSON output from this Python script and implement another tool to generate Wasm3 bindings

[huming2207]

Basically, we can either 1. modify this Python script, to make it generate Wasm3 binding instead of MicroPython binding, or 2. make use of the JSON output from this Python script and implement another tool to generate Wasm3 bindings

I think the second way is the right way to go. Because that LVGL's Python script is messed with a lot of template code for MicroPython and it's quite hard to modify. So maybe a separated tool that uses the JSON manifest output to generate a Wasm3 binding is better.

Here's an example of that JSON manifest:

lv_mpy_manifest.zip

[kisvegabor]

Happy to hear that you are interested! :slightly_smiling_face:

@vshymanskyy

I'm fond of LVGL, and I'm wondering if we could have a call / discussion of the vision, format and some technical details.

I think it'd be better to discuss it on GitHub to let others join the discussion.

@huming2207

From our side, both methods work. I agree that the JSON version is more independent and it's less likely that the wasm3 binding fails due to changes to the Micropython binding.

I hook in some of our developer who might be also interested: @embeddedt, @amirgon, @xiaoxiang781216

[huming2207]

@kisvegabor I'm thinking to implement a simple Nodejs tool with some template engines like Mustache.js, consume LVGL's JSON and output a generic C binding. I'll probably try doing it next week. But I may busy at work or preparing for some certification exams soon. So I don't know if I can finish it on time.

Meanwhile, two things we may need to be aware of:

1. Some host platforms like ESP32 does not have TCO support for now, so performance and RAM consumption overhead would be quite a bit if we do some complex UI logic (e.g. button or touch screen handling? UI content state management?)
2. I'm trying to add snprintf() support for formatting strings but I've got a strange memory corruption issue. I'm not sure if that's a typo in my PR or some other existing bugs in Wasm3. I don't know if that would affect the LVGL binding port as well, especially there's a lot of pointers passing around as I did in my PR. See here for more details: https://github.com/wasm3/wasm3/pull/216

[xiaoxiang781216]

Yes, we are very interested to setup a general binding infrastructure not only for micropython but also for other high level lanauage(e.g. JavaScript and WebAssembly). @no1wudi already made a prototype very similar to @huming2207 approach.

[kisvegabor]

I'm thinking to implement a simple Nodejs tool with some template engines like Mustache.js, consume LVGL's JSON and output a generic C binding. I'll probably try doing it next week. But I may busy at work or preparing for some certification exams soon. So I don't know if I can finish it on time.

We are not in hurry, take your time. :slightly_smiling_face:

Some host platforms like ESP32 does not have TCO support for now

What does TCO mean? Is it specific to wasm3 or something I should know? :sweat_smile:

I'm trying to add snprintf() support for formatting strings but I've got a strange memory corruption issue.

I'm not sure you can also use it but we added this printf lib in LVGL and it works pretty well.

Yes, we are very interested to setup a general binding infrastructure not only for micropython but also for other high level lanauage(e.g. JavaScript and WebAssembly).

If it helps and we find a reasonable way to do it we can restructure LVGL to put the JSON file in a common place where all the bindings can reach it. E.g. directly in the lvgl repo or in its own repo).

[vshymanskyy]

There are multiple things we may need to auto-generate. First, some C glue code - this will be a set of m3ApiRawWunctions on the Wasm3 host side. Second, we need to generate guest bindings. But there are multiple languages that may be targeting WebAssembly, so it looks like we'll have to produce bindings (imports and structures) for Assembly Script, TinyGo, Rust, C, ... . For some languages, object-oriented interface may be preferred (btw, does LVGL have suggested OO API?).

[vshymanskyy]

@huming2207 @kisvegabor sorry but both TCO and snprintf questions are a bit off-topic here, and not something we should bother about in context of LVGL.

[kisvegabor]

For some languages, object-oriented interface may be preferred (btw, does LVGL have suggested OO API?).

LVGL is written in C so it has no OO API, but it mimics it to the extent the language allows. E.g. Instead of btn1.set_width(20) we have lv_obj_set_width(btn1, 20). However, the Micropython bindings generates an OO API. E.g.

btn1 = lv.btn(parent_obj);
btn1.set_width(20);

We have an online simulator for the Micropython binding. It might be useful to see how the Micropython binding's API looks like: http://sim.lvgl.io/v7/micropython/ports/javascript/bundle_out/index.html

[vshymanskyy]

@kisvegabor I see quite a few ports of LVGL to webassembly. Maybe, some of these works could be reused?

• https://github.com/AppKaki/lvgl-wasm -> https://github.com/Rajssss/lvgl-wasm
• https://github.com/rafaelcaricio/lvgl-rs-wasm
• https://github.com/lvgl/lv_sim_emscripten

Conceptually, I'd love to see the same LVGL-based app (say, app.wasm) running in Wasm3 on desktop (sim?), in-browser, and on some embedded device. Desktop and Embedded cases are somewhat similar (in both cases LVGL is on the VM side, just the output device is different). For in-browser case, we can build LVGL as a separate wasm module, and link it to the app.wasm using some JS glue code.

[vshymanskyy]

However, the Micropython bindings generates an OO API.

Cool, I think this can be reused for the guest bindings then. It's worth targeting at least AssemblyScript, Rust, TinyGo

[kisvegabor]

I can talk about lv_sim_emscripten as it's created by @embeddedt (one of our core developers). It's not really a binding, but emscripten compiles the C code the web assembly directly using LLVM. But @embeddedt knows it much better than me.

Conceptually, I'd love to see the same LVGL-based app (say, app.wasm) running in Wasm3 on desktop (sim?), in-browser, and on some embedded device. Desktop and Embedded cases are somewhat similar (in both cases LVGL is on the VM side, just the output device is different).

That would be huge! Conquering the smartphones would be even better but with browser, we are actually already there :)

[MaxGraey]

Conceptually, I'd love to see the same LVGL-based app (say, app.wasm) running in Wasm3 on desktop (sim?), in-browser, and on some embedded device. Desktop and Embedded cases are somewhat similar (in both cases LVGL is on the VM side, just the output device is different). For in-browser case, we can build LVGL as a separate wasm module, and link it to the app.wasm using some JS glue code.

I know WASI will be move to WebGPU API direction. Browsers also will support it. So I'm wondering could LVGL will be mimic to WebGPU api as well?

API: https://github.com/webgpu-native/webgpu-headers/blob/main/webgpu.h And OpenGL backend: https://github.com/Michael-Lfx/google-dawn-mobile/tree/master/src/dawn_native/opengl

[kisvegabor]

I know WASI will be move to WebGPU API direction. Browsers also will support it. So I'm wondering could LVGL will be mimic to WebGPU api as well?

"Desktop-level" GPU work differently than embedded GPUs and LVGL was designed for embedded. However we see increasing usage of LVGL on high-performance devices as well so we are preparing the library to allow addend WebGPU, OpenGLES, or SDL support. It will take some time, but it's on the roadmap.

[MaxGraey]

Yeah, I mean create lightweight WebGPU ES which similar to OpenGL ES in term of adoption

[kisvegabor]

All look good so far. :slightly_smiling_face: What do you think, what is the next step?

[vshymanskyy]

I think the next step would be to create a minimal PoC, some "hello world" app with minimal API dependency. No need to auto-generate anything yet.

[huming2207]

I think the next step would be to create a minimal PoC, some "hello world" app with minimal API dependency. No need to auto-generate anything yet.

There you go:

And here's my SWD library binding demo, that the WASM binary calls the SWD API, reads a STM32's IDCODE and displays it:

[huming2207]

Yeah, I mean create lightweight WebGPU ES which similar to OpenGL ES in term of adoption

@MaxGraey This is a good idea, but I doubt it may not work well (i.e. achieve high FPS rate on something like STM32F4 or ESP32)

@huming2207 @kisvegabor sorry but both TCO and snprintf questions are a bit off-topic here, and not something we should bother about in context of LVGL.

@vshymanskyy indeed. But I think this may also cause some weird memory corruption issues and performance impact for UI logic implementations. But anyway, let's go ahead and try first.

[huming2207]

LVGL is written in C so it has no OO API, but it mimics it to the extent the language allows. E.g. Instead of btn1.set_width(20) we have lv_obj_set_width(btn1, 20). However, the Micropython bindings generates an OO API. E.g.

@kisvegabor I think for now let's just make a non-OO ish C API on the host runtime side, and generates a WASM side binding code (e.g. Rust or AssemblyScript) with OO API for users to consume it.

Yes, we are very interested to setup a general binding infrastructure not only for micropython but also for other high level lanauage(e.g. JavaScript and WebAssembly). @no1wudi already made a prototype very similar to @huming2207 approach.

@xiaoxiang781216 Any open source code available? If not then I will implement mine instead.

[vshymanskyy]

Conceptually, I'd love to see the same LVGL-based app (say, app.wasm) running in Wasm3 on desktop (sim?), in-browser, and on some embedded device. Desktop and Embedded cases are somewhat similar (in both cases LVGL is on the VM side, just the output device is different).

That would be huge! Conquering the smartphones would be even better but with browser, we are actually already there :)

@kisvegabor I envision something like this. Green boxes could be auto-generated:

This won't be Wasm3-centric, and can be extended to other runtime environments and languages.

We already have examples of "apps" in multiple languages: https://github.com/wasm3/wasm3-arduino/tree/main/examples_pio/Wasm_Advanced/wasm_apps They use simple Arduino-like API, but we also have this disclaimer:

WDYT?

There you go:

@huming2207 Awwwesome! :grin:

[no1wudi]

@kisvegabor @vshymanskyy I've made a simple prototype program here: https://github.com/no1wudi/binding

It uses TOML to describe interface definitions and can support the generation of global functions, global constants, and module definitions.

It's still in very early stage, maybe we can design a generic tool.

[xiaoxiang781216]

The tool will become more useful if we can make it generate not only LVGL binding for wasm3 but other C library for other high level language(much like IDL tool for proxy/stub).

[huming2207]

@kisvegabor I envision something like this. Green boxes could be auto-generated:

We already have examples of "apps" in multiple languages: https://github.com/wasm3/wasm3-arduino/tree/main/examples_pio/Wasm_Advanced/wasm_apps They use simple Arduino-like API, but we also have this disclaimer:

I think this is the right approach.

Keep in mind that my demo mentioned above is actually calling a dumb helper function that sets two texts into two lv_label instead of calling LVGL API directly. But I think it doesn't matter as it should work just fine anyway, unless something wrong with the runtime's memory management, or typos in the bindings.

[huming2207]

Here's a quick and dirty PoC script that generates C function signatures based on LVGL MicroPython binding generator's JSON manifest: https://github.com/huming2207/lvgl-wasm3-bindgen

Run npm i and then npm run dev and you should see some logs of the function signatures.

I've also attached a slightly modified gen_mpy.py script for your reference. Otherwise, we will see a lot of MicroPython-only types (e.g. NoneType instead pf void).

[huming2207]

The tool will become more useful if we can make it generate not only LVGL binding for wasm3 but other C library for other high level language(much like IDL tool for proxy/stub).

I had another idea about this. Ultimately, I think we should consider:

1. Similar to the idea of LVGL MicroPython binding generator script (the gen_mpy.py), we make a new manifest generator that make uses of pycparser and scan through the header files and generates a generic JSON manifest. This script is for C and C++ and it's project independent. That means, ideally, as long as the target project a valid C/C++ project, it should just eat in the C/C++ code and output a JSON manifest. This may be the final goal, for now we can just use the gen_mpy.py anyway.
2. We then create some different manifest consumers that consume the JSON manifest and generates the bindings for whatever you would like to. This is similar to my PoC project mentioned above. The manifest consumer is runtime-dependent, i.e. a manifest consumer for Wasm3, or another one for Wasmer, or another one for NAPI...but all of them can use the same JSON manifest from one manifest generator.
3. The JSON manifest between the generator and consumer is standardized so that it's easy for other people to implement a new consumer for a new runtime, or maintaining the current projects.

The advantages can be:

• Ideally, we can reuse the manifest generator for any C/C++ project, and include their manifest in their codebase for other manifest consumers to consume
• Manifest generator can run on CI/CD automation process
• We can reuse the manifest consumer for generating different binding code for different C/C++ projects

[no1wudi]

@huming2207 This tool should have two stage at least as you say, I recommend using TOML instead of JSON as the manifest:

1. TOML is more readable than JSON
2. TOML can represent hierarchical relationships clearly

For example:

[include]
system = ['stdio.h','alloca.h','string.h']

[global.function]
print="void printf(const char *)"

[global.const]
MAXJOBS=16

[console.function]
log='void printf(const char *)'
sum="int sum(int,int,int)"

[console.const]
INFO='Information'

[huming2207]

I recommend using TOML instead of JSON as the manifest

I think both of them are fine. It's mainly for the machine to read it, not humans anyway.

For the manifest generator, do you guys have some more ideas? I think I will have a try on this thing tonight: https://github.com/vickenty/lang-c

It's newer than pycparser and supports C11.

[kisvegabor]

@vshymanskyy

What a great progress, congrats! I also your approach. but I'm sure sure about one thing:

I envision something like this. Green boxes could be auto-generated:

In the embedded/desktop case m3ApiRawFunctions provides the LVGL API. In the HTML case, the app sees JS glue code's interface. So JS glue code will provide the same API as m3ApiRawFunctions?

Just to understand it clearly: how does it look like when a function of LVGL.wasm is called from APP.wasm?