WebGPU as low level graphics API

[olanod]

Soon will come the time to decide how to pain something on a screen, are there discussions about this topic in the community? didn't see any. Seeing what WebGPU is supposed to do seems like a natural fit for WASI to use this API instead of defining its own or doing something WASIVulkan/WASIMetal ... I imagine on top of, next to it, or even bellow, other APIs could be defined right? like stuff to draw 2d graphics web 2d canvas style, or create windows, or not sure if it would need to go lower level like defining a compositor, a framebuffer, etc.
I imagine Rust WebGPU implementation should make it easy for wasi runtimes like wasmtime or wasmer to come up with a working prototype since they are rust based, there are idiomatic rust wrappers, works natively on windows, mac and linux and there are a WebIDL definitions.

[sunfishcode]

I agree. As WASI grows, it's going to want a lot of the same functionality as the Web. And especially once we have webidl-bindings (or whatever that evolves into), there will be a straightforward way to directly use APIs like this.

WASI does have its own needs and use cases outside of the Web, and I expect it won't always make sense to use Web APIs. I haven't looked at WebGPU in detail myself, however graphics APIs are an area where using the Web API would have major advantages.

[sbc100]

Sounds plausible. I can also imagine direct Vulkan bindings being created.

Seems like it will depends on a couple of factors:

1. How much complexity would there be in the WebGPU -> Vulkan layer need to implement such a thing.
2. How much existing code out there is written for Vulkan vs WebGPU APIs.

[MarkMcCaskey]

I think one of the benefits of using some of the web technology is that they're the only, as far as I'm aware, ones thinking about untrusted use of the GPU. There have been some nasty security vulnerabilities with WebGL and one of Vulkan's selling points, as I understand it, is that it doesn't do runtime checking in release mode. I recall a number of years ago that malformed Vulkan without the validation layer could get the GPU into a bad state or crash the operating system or something in certain circumstances. Given that, I think exposing raw Vulkan will only make sense if the Vulkan drivers are built with malicious use in mind.

Vulkan seems like a good choice for building something on top of though!

More generally though, it's really tricky. Certain things like denial of service by asking the GPU to do a ton of work are still an issue with WebGL as far as I know.

I'm not an expert here by any means, but I am excited about getting graphical programs working with WASI!

[olanod]

@sbc100 1. I don't think there will be much complexity, I don't know either API but as I understand it WebGPU is being designed with heavy influence from Vulkan/Metal, Google and Mozilla. And the mentioned native rust implementation already works using dx12, vulkan or metal depending on the platform. 2. Of course code written for WebGPU is almost non existent since it's so fresh and still being defined but is totally worth considering it as the one backend to rule them all since it will rule the web for sure :sunglasses:

[aardappel]

It may be worth it to make the first WASI graphics API some kind of lowest common denominator API that works well on desktops, mobile, and the web alike, similar to OpenGL ES 3 / OpenGL 3 / WebGL 2. The point being that such an API would map relatively 1:1 onto these 3 types of platforms, and that a lot of code / game engines out there (in non-web land) already targets these APIs so would need few to no modifications to work (as opposed to WebGPU, which would require whole new engine ports).

Also note that WASI doesn't need to restrict itself to one graphics API, we can spec a WASI-Vulkan later, and/or WebGPU if it makes more sense. OpenGL may seem like an old-fashioned API, but it is easy to work with, and the amount of code out there that works with it dwarfs the other API options right now.

Once we have OpenGL, it may be possible to implement (a subset of) LibSDL to work on top of both the Posix and GL parts of WASI, further extending the amount of code that can run on top of this with little modification.

[programmerjake]

If we decide to supply OpenGL, I think providing it through EGL is probably best, since engines/applications can reuse their EGL code, we can follow a spec that's already been thought out and in wide deployment (Android, Linux, and more), and it allows relatively easy customization through EGL extensions. It additionally allows binding OpenGL, OpenGL ES, OpenVG, and more.

[tangmi]

I'd like to make another argument for WebGPU, which is a safe Metal-like modern graphics API that is being implemented in Rust with pluggable backends (as well as getting native support in browsers). This should mean that WASI could target Vulkan, DX12, DX11, Metal, OpenGL/ES, and WebGPU itself through a WebGPU interface.

It'd be lovely to see a WASI backend targeting the web and exposing a WebGPU-like interface via either native browser support or a polyfill via wgpu. It could mean truly "write once, run everywhere" (which hasn't always been true with regards to graphics, even with OpenGL).

Edit: Something we may also want to consider is how the graphics API will interact with composition APIs (e.g. DirectComposition on Windows, Core Animation on macOS/iOS, SurfaceFlinger/Control on Android, the DOM (?) on Web, etc.).

[doug-moen]

WebGPU is the only universal GPU API that is safe, portable, high level, high performance, and supports modern GPU features like compute shaders. Portable means: guarantees that code has the same behaviour on all platforms, backed by a conformance test suite. It works in web browsers, Windows, Mac, Linux, and mobile. Mozilla and Google are both implementing C library interfaces that target multiple platforms, and they have pledged to use a common WebGPU.h header file to promote portability. Mozilla's Rust library is layered on top of their C interface.

This looks like the next big thing. Game engines are paying attention. I'm planning to dump OpenGL and switch to WebGPU when it is ready, because I don't want to create my own abstraction layer that supports multiple back ends. OpenGL won't get me compute shaders on MacOS or the web. Vulcan will never run on the web. WebGPU is the only API that gives me compute shaders on the web, so why not just use it on all platforms?

[MarkMcCaskey]

Following up on WebGPU and OpenGL:

There's an interesting argument against OpenGL in the Glium post-mortem, namely that "Drivers are still crippled with bugs" [so abstracting over OpenGL in a cross-platform way isn't practical].

I've heard that WebGPU is more like Vulkan than it is like OpenGL; can anyone with context on WebGPU speak to this?

Security and being properly portable are my biggest concerns so WebGPU seems pretty appealing.

[tangmi]

I've heard that WebGPU is more like Vulkan than it is like OpenGL; can anyone with context on WebGPU speak to this?

WebGPU is the newest of the "modern" graphics APIs (e.g. Vulkan, DX12, Metal, Mantle), which I distinguish from the older APIs (OpenGL, DX11) as having explicit command queues opposed to implicit ones. As far as similarity, WebGPU compares closest to Metal (probably since Apple is the one that originally proposed it)--both don't require manual memory management while DX12 and Vulkan do. Like Metal (and OpenGL), I think WebGPU is a wonderfully friendly API that requires little boilerplate to be productive.

To me, the most compelling argument for WebGPU on portability is that it seems straightforward (although not trivial) to implement some standard OpenGL on top of WebGPU as a library (there's already a handful of OpenGL on Vulkan libraries, like GLOVE and Zink), while the opposite is neither straightforward nor trivial (especially given the problems addressed in the glium post-mortem).

[programmerjake]

To me, the most compelling argument for WebGPU on portability is that it seems straightforward (although not trivial) to implement some standard OpenGL on top of WebGPU as a library (there's already a handful of OpenGL on Vulkan libraries, like GLOVE and Zink), while the opposite is neither straightforward nor trivial (especially given the problems addressed in the glium post-mortem).

That's one of the major reasons we at (libre-riscv)[https://libre-riscv.org/3d_gpu/] are just writing a Vulkan driver and not an OpenGL driver for our open-source GPU.

[jpryne]

If someone is willing to layout the structure of a WebGPU-WASI implementation in Rust, and at least one or two example functions, I will devote my time to fleshing out remaining functions. IOW, plant the seed & I will water it.

[sunfishcode]

@jpryne An interesting thing about WebGPU is that it has a WebIDL API, meaning that once everything is ready we can use interface types to talk to it from wasm.

One of the first tasks for WASI is to evaluate that API to see if it's compatible with WASI -- that it doesn't make any assumptions about running inside a browser or having JS available, and that it fits within a Capability-based sandboxing model (roughly speaking, all functions should operate on a handle returned by the library, rather than having global impact).

If someone wants to start prototyping an implementation, one possible path is to hook up Rust WebGPU and the idiomatic Rust wrappers to the wasi-common standalone implementation of WASI in Rust. To reduce the amount of boilerplate needed, it may be desirable to build a tool to auto-generate some of the interface from the WebIDL source mentioned above, possibly with help from wasm-bindgen.

There's a lot in there, but feel free to ask questions!

[jpryne]

Thanks for your suggestions. I think, in the absence of examples to study, I should be starting with editing the documentation, (i.e. Todo: Update the IR Reference)

[bhack]

Is this only about GPU or could be related on any other kind of accelerator?

[doug-moen]

@bhack: The WebGPU standard only covers GPU. WebGPU corresponds to the intersection (common subset) of Vulkan, DX12 and Metal. Other kinds of accelerators (tensor processing units) are out of scope.

[tangmi]

If you have some time, @sunfishcode, help me understand what work needs to be done:

• API evaluation: I've looked at the WebGPU API and it seems compatible with how you've described WASI's sandboxing model. The entire API goes through a context object (Device) (sometimes indirectly through types that have a reference to the Device). As an implementation detail of wgpu, multiple instances of Device share the same gfx instance and resource registries (I think this just means two devices in the same process can't create resources with the same identifier?), but I dont think this intereferes with WASI's sandboxing.
• hostcalls: You mentioned boilerplate code needing the be written (or generated) in wasi-common. Are you referring to the hostcalls macro invokations in wasi_common::hostcalls::*?
• Windowing: wgpu needs a window context to present to, but I'm unsure how that should be exposed to WASI. I think for WASI to be successful at filling the 2D application niche, it needs to support more complex windowing APIs (i.e. OS composition layers), but for this work maybe it's fine to define a simple "create this window and wgpu device"?

Sorry if this is a little disorganized--I'm trying to wrap my head around how all the pieces fit together. From my understanding, one needs to:

1. Write (or create a tool to generate) a file at wasi_common::hostcalls::webgpu, containing essentially wgpu-native modified to input/return WASI types.
2. Implement wasi_common::hostcalls_impl::webgpu that mostly forwards to wgpu-native or wgpu-rs.
3. Have wasi_common::hostcalls_impl::webgpu call to wasi_common::sys::*::hostcalls_impl::webgpu to handle platform-specific stuff (device creation/window creation?)
4. Add the webgpu api to rust-wasi (?) so that users can consume the API (e.g. wasi-misc-tests?)

Please let me know where I'm hand-waving too much! If I can understand what needs to be done, I'm interesting in taking a shot at it!

[sunfishcode]

I apologize; I somehow missed your message earlier. Some pointers:

• There's been some preliminary discussion of Windowing APIs here.
• And yes, the rust-wasi crate, since renamed to just wasi, would be a good place to host Rust API bindings for this.

[AldaronLau]

@sunfishcode Has anyone attempted to start a specification for a Windowing / WebGPU / Framebuffer module? I'd be interested in contributing in this area, but I don't want to do duplicate work.

[caspervonb]

I've been silently working on a separate thing dubbed WAKI (WebAssembly Khronos Interface). Basically it's low level bindings to Khronos Group APIs, the windowing API is somewhat derived from the now defunkt OpenKODE API; Graphics from OpenGL and Vulkan, Audio a mix of OpenSL and OpenAL.

I don't have an implementation I want to share at the moment as it's very rough, I've been busy with implementing deno, deno-wasi and wasi-test.

Ultimately, I think there are more fundamental things we need to iron out in WASI first before we try tacking on multimedia APIs like what is the new modularisation going to look like, threads, sockets, dynamic loading, etc which is why I haven't brought it to WASI (also I can just do whatever the heck i want while iterating on it, which is faster 😉 )

[AldaronLau]

@caspervonb Good to know! Thanks.

[sbc100]

I've been silently working on a separate thing dubbed WAKI (WebAssembly Khronos Interface). Basically it's low level bindings to Khronos Group APIs, the windowing API is somewhat derived from the now defunkt OpenKODE API; Graphics from OpenGL and Vulkan, Audio a mix of OpenSL and OpenAL.

Oh my, the mention of OpenKODE certainly brings back memories! I was involved with that at the time too. Certainly a lot of overlap with what wasi is doing today.

[jpryne]

"I've been silently working on a separate thing dubbed WAKI (WebAssembly Khronos Interface)..."

Btw, "waki" is a Japanese word meaning "sword", (as in "wakizashi",) along with other things. Let me know if you'd like a logo...

[Kagetsuki]

"I've been silently working on a separate thing dubbed WAKI (WebAssembly Khronos Interface)..."

Btw, "waki" is a Japanese word meaning "sword", (as in "wakizashi",) along with other things. Let me know if you'd like a logo...

Waki is arm pit. Wakizashi is 脇差, which is 腋 "arm pit" and 差 which means like to put something through something and hold it there (like hold by sticking through a belt). The short sword known as wakizashi is just how it's called because it was a short sword designed to be fastened under your arm/arm pit.

[trusktr]

Here is a TypeScript program compiled to WebAssembly with AssemblyScript, using WebGL APIs outside of a browser in Node.js (the JS glue code calls the webgl-raub package):

https://user-images.githubusercontent.com/297678/107162186-5f2fbc80-6956-11eb-90d1-13d351fb5c55.mp4

The WebGL code is written in AssemblyScript, and currently starts like this:

import {WebGLRenderingContext} from '../node_modules/aswebglue/src/WebGL'

const gl = new WebGLRenderingContext(...)

// call gl.whatever as you would in JS/TS

This means we at least have a WebGL interface in AssemblyScript currently working in Node or browsers.

Next we would need to make the WebGL (or WebGPU) interface in WASI, but the AS code would remain the same.

[trusktr]

Oops, I forgot links!

ASWebGLue: https://github.com/battlelinegames/ASWebGLue

LUME Glas (port of Three.js to AssemblyScript that will likely fork direction): https://github.com/lume/glas (part of the LUME project, http://github.com/lume/lume)

AssemblyScript's #gamedev Discord channel where we chat about ASWebGLue and AS game dev: https://discord.gg/A5n6qdeYfR

LUME's #glas channel where we talk about ASWebGLue and Glas: https://discord.gg/6XvnkMb

The Node-AssemblyScript-WebGL example isn't pushed up yet, but will be soon...

[freedaun]

I came here disappointed that WASI doesn't allow me to paint anything to a bare rectangle of pixels. Having a pixel buffer means something like a browser could (theoretically) be created.

I am NOT asking for fonts, graphics primitives, and I certainly do not support the suggestions I see above, which would make WASI into an extension of some other project. No, just a pixel buffer and the ability to hand it off to the system. Something obviously secure. Something obviously compatible with any of the options above. The thing even DOS could deliver.

[lachlansneff]

A regular pixel buffer with no gpu acceleration is not obviously useful for anything aside from toy projects.

[freedaun]

Indeed. So sadly we are limited to toy projects like:

• 2D games, like Patience and Minesweeper
• browsers
• Slack
• MS Office

Mostly useless crap. Unless projected into Virtual Reality. So just another bad idea. The original poster was trolling also, asking for 2D. Obviously not what people want.

[remexre]

Browsers, Slack, and I assume Office are all GPU-accelerated these days; it's really hard to not kill battery life while drawing to a hidpi screen without the GPU.