[FEATURE] Support SkiaSharp as a Blazor Extension

[RChrisCoble]

My understanding is SkiaSharp running on WebAssembly is in progress with a target release of May. This would allow a "Blazor Client" application to use SkiaSharp and Skia fully running in a supported client browser.

One highly desired feature after that would be supporting the Blazor model of having the option to run the business logic calling SkiaSharp either client or server via a "Blazor Extension". Running from the server is important for scenarios of smaller download size, thin client support, taking advantage of server capabilities, etc. I understand the performance would be quite different in the server scenario, but it would still 'work'.

An example of this concept used on an HTML5 Canvas API is here: https://github.com/BlazorExtensions/Canvas

This isn't rocket science, as the HTML5 canvas is a known API existing in the client browser, and the Blazor Extension wraps the canvas API so when running on the client, the extension simply calls the canvas API directly. When running on the server, the Extension marshals the API call to the client over the Blazor SignalR connection. There's some extra magic in that linked Extension to batch calls for performance, but you get the idea.

While I'm not an expert on the SkiaSharp architecture, my assumption is the only way this would work is have Skia still run in the client browser, and SkiaSharp (or a thin wrapper) run on the server to marshal the calls over SignalR. Alternatively (but probably impossible for this repo) is have Skia marshall the calls over SignalR and have them execute using WebGL or Canvas.

So, ultimately the feature request is once SkiaSharp can run in Client Blazor, please support SkiaSharp as a Blazor Extension.

[mattleibow]

https://github.com/mono/SkiaSharp/issues/1219#issuecomment-611157056

From @RChrisCoble:

Regarding:

[FEATURE] Support SkiaSharp as a Blazor Extension (#1194) This is what we are going to be doing here.

When it's a Blazor Extension you can utilize the SkiaSharp API on the Client or Server Blazor deployment model. I believe what you are doing here is akin to the forked Uno repo, allowing Skia/SkiaSharp to work properly when running under WASM in a client browser. My specific feature request around making it a Blazor Extension was to also allow the API to run on the server with the rendering happening on the client.

This would most likely mean Skia (and maybe SkiaSharp) would need to be running in the Client Browser, with the SkiaSharp API calls being packaged up and marshaled over the Blazor SignalR connection when the Blazor Extension was running on the server.

An example of this was done with the HTML5 Canvas here: https://github.com/BlazorExtensions/Canvas

If you look at that GitHub example, they wrote a wrapper around the HTML5 canvas API's which ended up doing one of the following: (1) If the Blazor Extension was running in a client browser, issued the draw call directly or (2) If the Blazor Extension was running on the server, marshaled the call over the SignalR connection to the client to be executed.

The Blazor server deployment model is valuable in situations where you can't deploy multiple MB's of DLL's down to the client just to draw anything. (with that multiple MB's NOT being Skia/SkiaSharp, but the business logic using the SkiSharp API.)

[mattleibow]

Seems that we might need to wait for https://github.com/dotnet/aspnetcore/issues/5466 to be completed so we can link SkiaSharp with the WAM part. Not sure about dynamic linking...

[RChrisCoble]

[EDIT]
I see this was posted under the Blazor Extension feature and not under the WASM feature. So I changed my comment here.

@mattleibow Does this mean moving forward with running SkiaSharp in a client browser under WASM is moving forward Ok, but running it as a Blazor Extension (meaning I could run it on the server as well) needs to wait on #5466

I'm hoping it's the latter. If we could run SkiaSharp directly in the client browser still that would be fine.

[mattleibow]

@RChrisCoble, no prob!

This is a mix of issues/features. Let me break it down and then maybe you can see where exactly you are needing a feature or if we need to focus on some things.

First, the real basics, SkiaSharp obviously works just fine on the server as a function/API that you can use as an endpoint for the basic <img src="/path/to/generated/image" />. This is pure server code that uses basic HTML to get the image. I assume this is not what anyone is talking about, but just for general clarity.

For the WASM/client code, this is actually 2 separate cases: Uno or Blazor.

Uno is working just fine and this is currently what the #1333 PR is for. It adds a Uno view and runs SkiaSharp right in the browser today. When ready, this will just be a matter of adding the view to the XAML/code behind and then the browser will draw it for you. Currently, Uno takes advantage of WSL to effectively build your app on Linux under the hood and produce the working website.

Blazor is a bit different as it does not yet support the feature to statically link the app with SkiaSharp libraries. This is what https://github.com/dotnet/aspnetcore/issues/5466 is all about - basically add the features to compile the WASM bits with static libraries. It appears (at the time of writing) that we have to wait for .NET 6 next year or so before we get this feature.

The difference between Uno and Blazor is the app model. Uno is basically UWP apps, but in the browser. Blazor is an ASP.NET Core Razor website, but in the browser.

[mattleibow]

As I write the comment above, there is the fact that Blazor also has the server components. Now, that might just be possible today because the code actually runs on the server and new png/jpg files are sent to the client. I have not investigated this case at this time, but might be useful to look into if it actually helps in the case what you are looking for. I am assuming you are more wanting the code to actually run on the client browser.

[RChrisCoble]

@mattleibow Thanks for that detailed description, that helped me considerably. Now I understand how Uno is able to statically link to make this work, compared to Blazor which needs AOT support which was dropped from .Net 5.

We have a C#/.Net engine we wanted to run in the client browser directly, using SkiaSharp to render. That would have been the ideal solution for speed/performance. It sounds like the only avenue to that end right now is leveraging Uno/SkiaSharp which we'll have to evaluate once #1333 PR comes in.

That being said, this engine participates in a framework of other components that often have to run on the server right now because of how they were built. If we could use a SkiaSharp "Blazor Extension" that only supports server side execution (for now), with the forward looking statement that the extension would someday be supported using Client WASM enabled via .Net 6 AOT compilation, that would be valuable.

[EydenJones]

So does mean that being able to draw using skiasharp in a client side blazor app isn't going to happen until late 2021? (release date .net 6 which will include AOT compilation) :-(

[mattleibow]

Unfortunately, that looks to be the case, but I can assure you the team is working on this as fast as they can. I'm in the chats and I see all the good words being thrown around.

Not sure when exactly things will go out, but hopefully sooner than later - even if a preview. I think previews usually start early in the year, so it is quite soon.

[Flippsor]

@mattleibow great work so far! Sad to hear that we have to wait for AOT compilation from mono. I already tried the Canvas/WebGl Blazor extension, but it is not very performant due to JS Interop calls. I also tried the .NET binding for WEBL through Webassembly (https://github.com/WaveEngine/WebGL.NET) in Blazor and got it working (see screenshot). Not quite sure how they do the binding to Webassembly in detail, but maybe it is possible with Skiasharp, too?

Edit: Maybe Webassembly.Bindings can help?

[mattleibow]

I had a look and things still have a sad thing. As you saw, going from .NET to JS to C++ is not the best. Especially with the number of calls being made. The WebGL part is not too bad as it is just talking directly to JS. Unfortunately, the libSkiaSharp lives in C++, so we have to hop the bridge twice.

Thinking wildly here, it might be possible to create a Blazor app, and then use Uno as a simple container to hose SkiaSharp things. Like an iframe or maybe set up some communication between the two. The main app can be Blazor, and the drawing part can be an iframe. This obviously depends on the use case... Sort of treat the Uno part as a separate "canvas"...

[mattleibow]

I made some interesting thing: an Uno iframe inside a Blazor app. The communication is a bit dodgy right now, but might not ne too bad depending on the use case.

https://github.com/mattleibow/SkiaSharpUnoBlazorApp

[Flippsor]

This looks really great! Already checked it out and got it working. Investigating the code, i recognized that the communication is done through the iFrame via JS Interop. This will be a little bit tricky because we are planning to create an editor where the same objects are used in blazor lists and also drawn onto canvas. So as i understand, using the same object reference in blazor & SkiaSharp will only be possible when AOT compilation arrives?

[mattleibow]

Sadly yes. Right now, it is 2 entirely different runtimes and websites, so there is no way to reuse anything.

But, I want to try out a different communication model. Using the messaging. This should allow me to use json objects. But, this will still go out via js.

[mattleibow]

Wait a sec... I wonder if there is a way to pin memory in js and use that in both sides... I do this, sort of, for skia... I create the pixels and then pass the pointer to js, might be able to read it in uno/blazor

edit

That will only ever work for blittable structs. Not useful for general classes.

[mattleibow]

I did an improvement with the communication. No longer using slow fragments, but direct JS.

[mattleibow]

I have been hacking away on this, and I think I may be able to do something for Blazor server-side: https://github.com/mattleibow/SkiaSharpBlazorComponents

<div>
    <SkiaSharpImage OnPaintImage="@PaintImage" />
</div>

@{
    void PaintImage(PaintImageEventArgs e)
    {
        var canvas = e.Surface.Canvas;

        canvas.Clear(SKColors.Blue);
    }
}

[Flippsor]

Great work!

Afaik the API for Blazor Server and Blazor Webassembly for Razor components is identical. So people could start with Blazor Server + Skia and then switch to Blazor Webassembly + Skia when the AOT is ready.

[mattleibow]

That is the plan!

But the client wasm one also is a bit different with the fact that I can have a canvas view...

I need to investigate if we should

• use an image for the server and a canvas with a real time render
• use an image everywhere, and have an additional option for the client...

I am very far from the web space, I will need some help 😅

[RChrisCoble]

I would think the best you can get here is drawing an image on the server and having that sync across using the Blazor SignalR connection.

For the client, as long as the draw calls are not going over the JS/Interop boundary to slow it down, whatever solution should work.

[matdennoigi]

@mattleibow Your code only works on Blazor Server model, not work on WASM, because your SkiaSharp version that you used for code can't work on WASM. Actually, your Skia rendering was excuted on server then passed to client through HTTP.

[GeorgeS2019]

@mattleibow based on the SkiaSharpBlazorComponents example, how would you improve the code to make SkiaSharp animation possible.

[mattleibow]

Animation? That is hard. Because only the server can render, the animation might have to be pre-rendered. Not sure if that is going to be nice with latency.

We really need the ability to link native libraries with Blazor. Like reeeeeeaaaaly need it. The only thing that is possible to do now is render a few png and then fake it. But that is not good at all.

[mattleibow]

The only thing that can work for client rendering is an embedded Uno app.

[GeorgeS2019]

@mattleibow have you tried checking the animation performance with embedded Uno app? Perhaps the combination of client side embedded Uno and server side image rendering may be a good use case, during the transition to AOT in the coming .NET6

[mattleibow]

The embedded Uno app should run at native performance of AOT WASM in an iframe.

I have not done any actual tests, but I was able to get 60fps with a flappy bird game, and that should not be affected by an iframe. And, that game was also just the interpreter.

[GeorgeS2019]

@mattleibow it takes around 1 min or more to compile the SkiaSharpUnoApp alone through WSL2.

• Do you have suggestions what I have done wrong or what I could improve that.

The project works. I still try to understand what "Hack" you used to make embedding UnoApp works within Blazor. I compare that with the https://github.com/unoplatform/Uno.SkiaSharp/tree/uno/samples. It seems there is lot of copying the compile output files from the SkiaSharpUnoApp to that of the SkiaSharpUnoBlazorApp.

I do not have sufficient experience with Uno.SkiaSharp. It is unclear how much I could bring the Unit Tests in Uno.SkiaSharp into the SkiaSharpUnoApp and SkiaSharpUnoBlazorApp project setup you have implemented.

I hope you will continue your investigation and perhaps we will see your work in coming community asp.net standout or coming Microsoft events.

I also hope the discussion here will speed up the implementation of AOT that you see as key to bringing SkiaSharp to Blazor.

[mattleibow]

Another related issue for getting better WASM support: https://github.com/dotnet/runtime/issues/44636

[GeorgeS2019]

@mattleibow thanks for the tip. I will follow up

[RChrisCoble]

Hello @mattleibow, have you thought any further on how SkiaSharp might run as a Blazor Server module?

Option 1: Load Skia and SkiaSharp in the browser. Abstraction layer sits on top of SkiaSharp API’s on server and marshals draw calls to client. Option 2: Load Skia only in the browser. SkiaSharp maintains same surface API and runs on the server, but marshals calls to the remote Skia running in the client browser.

[RChrisCoble]

Hello @mattleibow.

My suggestion is a stepwise approach while we wait for Blazor AOT to come into place:

• Start a POC for a SkiaSharp Blazor Server extension leveraging Uno’s SkiaSharp client rendering support. - This might be as simple as just drawing a rectangle from the server to the client.
• Assuming this works, expand out the support with additional SkiaSharp API calls.
• As .Net 6 Beta AOT support starts to roll out, update the POC to support SkiaSharp rendering in the client and remove the Uno implementation.
• Harden the implementation for a PR back into the SkiaSharp repo.