Add non-web implementations

[LifeIsStrange]

I know this repository was originally intended for measuring js frameworks, and it has been a tremendous success. However @krausest , hear me out:

Performance metrics such as FPS/frame time, first time to paint, CPU/RAM use and energy consumption are essential. Chromium, gecko, GTK, QT, flutter and others all compete regarding performance and energy efficiency. Nowadays it is impossible to find even simple comparisions of GUI libraries rendering performance, when science is absent it only leave space for myths. If the project accepted non-web implementations, it would enable for the first time comparison with QT, flutter, cocoa, etc The simplest implementation and HIGH IMPACT one would be a direct port of the existing react implementation into a react-native desktop equivalent (almost the same code, but different renderer) Spoiler, contrary to popular belief I expect the web versions to perform better, especially newer frameworks such as Solid and svelte. note: my point about including other performance and energy consumption metrics is completely optional, my main point is to allow non-web implementations. Of course they should be displayed with a different color/signaletic in the results UX. If some non-web GUI libraries are faster than the current web SOTA then it would give us a positive information about how much performance gain we might expect via future frameworks and chromium (or gecko) optimizations. On the other hand it might (and will for some) reveal that some GUI libraries significantly underperform.

@krausest plz make this happen, you can make the world a better place! Or at least allow the community to submit implementations :) @ryansolid friendly ping, thought it might have picked your interest

[LifeIsStrange]

Bonus, optional comment, feel free to ignore :) See also my rant about GUI rendering libraries performance myths:

Here's my (unfortunately non-mainstream) semi-expert opinion on the topic:

The myth of web vs native performance

Being native is a non-criteria

In the minds of many devs, when they hear native, they hear "better" or "faster". In essence for a UX library to be native is an ad-hoc, contigent property. It is native if and only if it is shipped by default on the OS. As such, since OSes now ships with webviews by default, one could argues webviews (although not electron) are native. If by native we instead means the officially recommended libraries then chromium shall be considered native on chrome OS but more importantly being native is a cutlural phenomenon of dev politics and NIH syndrome, it is not in any way a technical critera and therefore has zero explanative power regarding the rendering performance of said library.

In addition to this common misconception, a potent psychological factor that may have made people believe that the web is slower, is because the web usually involve internet and loading things indeed is much slower than having them directly in disk or in RAM. This however is contingent since electron and Ionic can have local resources, even browsers have localstorage and various cache APIs and filesystem APIs.

Finally the myth has some grounding in a few low quality scientific papers that evaluated browser performance vs native on android smartphones circa 2010. This evidence was unextensive and more importantly it was 12 years ago, since then things have significantly changed yet there are no recent decent scientific paper on the topic. BTW as a reminder, flutter is not native.

2D renderers are the backbone of an UX library

As a reminder native UI libraries have chromified with time. The android native renderer (used by RN) and flutter have switched to Skia as their 2D renderer. Skia has originally been evolved by chromium devs for chromium needs (although it now is becoming more inter-project), e.g. its versioning numbers match chromium release numbers. Skia, the originally-web renderer is the fastest 2D renderer out there (destroy Cairo (GTK), the QT renderer, WPF, etc). One reason for it is that it is the renderer that has received the most human resources (commit count) enabling e.g. vulkan support, another is that it is one of the few to decently leverage the GPU for 2D, finally because of its ubiquitousness it has received contributions from hardware makers (e.g. NV_PATH) The only contenders worth mentioning are promising but abandoned scholar projects (fastuidraw, pathfinder) except for webrender, which is made by mozilla and is competitive. Honorable mention for the promising but still in development piet-gpu renderer.

Therefore, based on my researches, a UX library is necessarily slower than chromium for rendering tasks if it is not based on Skia or webrender (no native library currently leverage webrender, maybe because of its name) Android native, react native, flutter, jetpack compose, chromium and one new microsoft ux library (was it maui?) share this in common hence they have the same backbone and main rendering performance bottleneck.

the future of 2D renderers

2D renderers could significantly outperforms skia, there are many missed opportunities of optimization and hence non-web libraries could outperform chromium in the future however no-one care. Next gen 2D renderers needs 2 things: brilliant devs in optimization/rendering/math (rare) and human resources (funding). The thing is: future 2D renderers will never come true because new UX libraries never allocate resources to their renderer, they do not have the means or the competence and this has been true since the beginnings of time.

the performance of an UX library does not only depends on its renderer

the renderer is only one piece in the puzzle. An UX library can have many different kinds of optimizations:

1. an incredible variety of caches (increase ram and vram use but is essential)
2. a retained mode architecture, which significantly outperforms immediate mode libraries, especially for energy consumption
3. an efficient scrolling engine
4. a multithreaded architecture
5. optimizations arround clipping/pixel occlusion
6. SIMD
7. smart algorithms about style/layout/painting invalidation
8. how it configures its 2D renderer
9. many miscellaneous optimizations such as e.g direct compositing and a thousands more

About point 8) > how it configures its 2D renderer 1} As stated above for most of its history, Skia has been finetuned for chromium and chromium finetuned for skia. This is a theoretical major advantage. The android native renderer (which is abandonware BTW) has recently switched to Skia (it was not before) and I expect it to not leverage new skia features/optimization flags (there must be a great cultural and human resources inertia) 2} However flutter does enable new Skia provided optimizations apparently at a quicker pace than chromium (chromium is more conservative regarding skia flags). I expect this to provide ~20% more performance for flutter vs chromium only regarding their use of skia (1} is unquantifiable though)

regarding optmizations cited previously (1) to 9)) I expect chromium to have a very significant advantage vs flutter (and even more so vs android native renderer..) The reasons being:

1. I know by expertise that chromium has 1) to 9) implemented. As for flutter those informations are hard to find but there are probably many lacking (although some are confirmed, such as a retained mode architecture via display lists)
2. The amount of optimizations you can implement is directly proportionate to the human resources allocated to the project. Chromium has 1000000 commits, flutter has 27000, the difference is humiliating. It approximately means that chromium could have rewritten from scratch flutter 37 times consecutively. While this reasoning is heuristic, it is essentially the cases that chromium has received more optimizations than flutter. Android native is as said previously in an abandonware/maintenance mode state and hence become increasingly obscolete as time passe although jetpack compose (but not react native) might become competitive with flutter regarding that aspect.

the future and present of web performance

There has been recent breakthroughs in web performance:

1. The CSS containment API is revolutionnary and enable to easily enforce guarantees about layout (no overflows/invariants). This eliminate many pathological cases of cache invalidation, in fact in real world cases it has been found to yield 1000X perf improvements. There are other APIs though e.g. such as will-change.
2. Ionic capacitor fix a performance flaw that was in cordova regarding the loading of native plugins.
3. web frameworks are becoming much faster Vue.js 3.0 has become disruptive and is ~2X faster than angular and react. React lags behind although the next release of react should reduce the gap via e.g. https://reactjs.org/docs/concurrent-mode-intro.html However it doesn't stop there, VDOMs are now considered obscolete and DOM compilers such as https://www.solidjs.com/ achieve much better performance. source

Hybrid web apps can interop with native code trivially

People often believe that web apps can't leverage native code however that is totally wrong. Sure that is the case for PWAs but not for TWAs and cordova/capacitor derivatives, the same way as Electron allows to call native code.

1. there is a large ecosystem of plugins 6454, in fact it is my (unverified) belief that there are more Ionic plugins out there than react native/flutter plugins. In addition to the Java/Kotlin interop, Nativescript is also a great development https://nativescript.org/ Kotlin Multiplatform also enable boilerplate-less native code interop.

what a hybrid web app cannot do

A hybrid web app can do anything, it can call any native APIs such as e.g CUDA for machine learning. The only thing it cannot do is calling 3D native rendering APIs, which means that if webGL 2 or webGPU is not enough and you really need Vulkan (e.g for performance critical video games) then yes a native app can make more sense and outperform a hybrid web app.

the myth that hybrid web apps are unpopular

Hybrid web app are extremely popular, in fact cordova/Ionic have more marketshare than React Native + Flutter combined. And I'm not even talking about the fact that more people use their smartphone browser than their desktop browser.

Nobody cares and yet everyone is impacted

Nobody cares about graphics library performace. Usually developpers have close to zero knowledge regarding the intricacies of the engineering behing UX libraries. Nobody (no scientist, no engineer, no end user) care to do relevant benchmarks of isomorphic apps test-cases. That is why, the myth that web performance is slower than native frameworks persists. Although I expect this belief to eventually die (for instance, people are slowly realizing that VScode is the fastest IDE because it is based on electron). Also electron has taken over the desktop and had we the belied that web is slower, that belief would be now irrelevant since smartphones are becoming faster than many laptops. Anyone that does a quick glance at flutter issues on github shall quickly see that is has many relevant pathological performance issues.

Nobody cares yet it impact every end user: Sure apps are OK but excellence is about reducing frustrations and increasing confort by increasing meaningful performance metrics such as FPS, first time to paint, etc Secondly energy consumption matters.

Nobody cares yet it impacts many smartphone developers. People sacrifice the vastly superior expressivity and maturity of the web for inferior ecosystems (dart) because of The Myth. React native existence itself is only based because of the myth. Despite this, I assert that most react native apps would experience a free performance gains (beyond expressivity and confort) by migrating their codebase to ionic react.

To conclude, despite all the evidence I provided, at the end of the day empirism would be welcome and as such, we need to start building a set of useful benchmarks that would be cross-GUI libraries and as such enable raw performance comparisons for the first time, and secondly, enabling to catch regressions in library updates (chromium, flutter, etc have potentially major uncatched performances regressions at each release)

[krausest]

That would be really interesting. But I'd rather put it into a seperate repo.

Some questions would have to be answered:

• What form/layout are we measuring? I'd say lots of rows in a vbox(layout). Each row has a label, an input field and a delete button
• What operations are we measuring? Anything with CPU throttling is not an option, so create and clear might be good candidates.
• What rules for the look and feel do apply? Forcing all implementations to create the same look might be lot of work or impossible. Maybe it's easier the other way round: Use a native look and feel and create a companion html page that mimics the look and benchmark against that.
• What programming style / model should be required? I'd say let's start with imperative programming models (it's a browser vs native rendering benchmark, not a imperative vs. model driven benchmark).
• What metrics should be used? CPU duration might be feasible. I don't know if I could measure memory in a correct way (rather not virtual memory, but working mem or physical memory. Should we count all processes?)

If you have a prototype for a QT application (or something else that might compile rather easily) that creates a few rows please let me know and we'd use that as a starting point.

[hinell]

Well it would certainly require a separate repot to produce metrics. Thsi would make a lot of efforts for sure. On the other hand, I'm sure it makes little sense to test apps in QT where basicallh the same Blink/Webkit engine is used. I bet the performance would be the same.

[ClassicOldSong]

At this point I think this repo should be called web-framework-benchmark. As JS is now running everywhere, there're platforms that can run some JS frameworks but they're not web, like React/Vue/Angular implementations on NativeScript. Some frameworks achieve high mark in this benchmark by leveraging HTML features like using innerHTML for creating static content, which means they're not portable to other environments that don't have native innerHTML implementation, and probably not that performative.

[LifeIsStrange]

There is no reason to artificially reduce the scope of this benchmark, as I extensively argued, there is a crucial unmet need of fair comparison between identic GUIs on different GUI libraries, regarding metrics such as performance (and ideally energy consumption, and fine grained metrics (first time to paint, etc)). If this benchmark should be renamed into something it would be gui-framework-benchmark. It's right that currently no one bothered to implement native (and non-native) gui libraries in this benchmark but it will come eventually. I personally aim to implement it myself for react-native-desktop when i'll find the time, unless someone does it before me.

[ClassicOldSong]

I'm not saying limiting this benchmark, but it is now limited to web as a fact. Though expanding this bench to a wider range is a good wish, it's hard to find a common ground to measure all these variables fair enough.

Regarding to those myths you mentioned, me myself just feel off for web pages / self paint than native. Text rendering usually have a special texture with skia based renderers, I couldn't tell why. Flutter isn't "native" as well, it lags and jitters, and can't catch up with native feature changes(like the ripple effect change on android 12), while Qt just can't handle macOS UI animations correctly.

Though web is getting faster and faster, it still differs from true native in some way, and someone like me do care about those differences.