Render to OpenGL Target

[J-Cake]

I've been trying to do this for a while now - I've got an OpenGL Context with glfwJS created, and I'd like to use Skia to draw to it, I'm just wondering if there's a way to make the two interoperable? I realise there's a lot to bridge, but according to this post, it seems doable.

[J-Cake]

Further digging has revealed it's fairly doable, although I'm not sure how it would be integrated with NodeJS:

https://gist.github.com/ad8e/dd150b775ae6aa4d5cf1a092e4713add#file-glfw_ship-cpp-L108

[nornagon]

Just dropping in to note my interest in this, it would be great to be able to use skia in js to render directly to a window!

[konsumer]

I am also very interested in this. Has anyone figured it out?

[lucasmerlin]

Skia seems to have some slight differences when rendering on the cpu vs the gpu and I need to perfectly match how a browser would render the image. So when I'm rendering with skia-canvas, it would be great for skia-canvas to support rendering on the gpu! I saw that there is a gpu-render branch and I was able to modify it to use vulkan to be able to run it in a docker container: https://github.com/lucasmerlin/skia-canvas/tree/gpu-render Using this in production right now and it's working great so far! I can also share some more details of how I was able to get this to run in docker, if anyone is curious about that.

Are there any plans to actually release a version with gpu support to npm?

[samizdatco]

I've been able to get things working well on the Mac (both for drawing to windows and running offscreen with the GPU renderer). The reason I've been wary of trying to integrate the changes is two-fold:

1. It's unclear to me whether it will be possible to ship a single version of the library that switches between CPU and GPU modes in a reliable way (and without bloating either the size or the system requirements)
2. I'm totally out-of-touch with how OpenGL works on Windows and Linux at this point.

It's encouraging to hear you've been able to get the GPU renderer working in Docker containers, @lucasmerlin! In your fork it looks like you're using the Vulkan backend. I'd been thinking of sticking with OpenGL since it stood the best chance of running on all three supported platforms (and using something like glutin for windowing). Can you explain a bit about how Gl/Vulkan/Mesa and X11/Wayland interact? It's unclear to me what libraries need to be accounted for.

I'd also be curious to see some Dockerfiles with the necessary dependencies for a) offscreen GPU rendering, and b) opening a window via glutin or some other cross-platform abstraction.

And then there's Windows. It'd be great if someone could test out whether using glutin for windowing and GL for rendering is a reasonable option there as well.

[konsumer]

Let me know what I can do to help. I am strong on the js-side, weak on the rs-side, and am happy to help with anything I can.

[samizdatco]

Building off of @lucasmerlin's fine work, I've gotten GPU-based rendering working on macOS (using Metal) and Linux (using Vulkan). My understanding is that the Vulkan backend should work on Windows as well, but the VMs I've tested it on don't have GPUs. So I could use some help verifying whether it's functioning properly from someone with an actual Windows box to test on. I'm also curious to know what additional dependencies are needed to get this version running in common Docker containers.

There's a new .gpu attribute on the Canvas object which will be true if it was able to set up a hardware-based renderer (and can be set to false to switch back to the CPU renderer manually). Before merging the pull request I'd love it if people could kick the tires and let me know if they find any regressions compared to the software renderer. As far as I can tell though, the changes are nearly all positive and go a long way toward addressing #88, #85, and #38.

You can install the current release candidate (rc3) via:

npm install https://github.com/samizdatco/skia-canvas#gpu

[lucasmerlin]

I'm using the following build instructions to build the gpu docker image:

image: "renderer"
from: nvidia/vulkan:1.2.133-450
docker:
  ENTRYPOINT: 'sh /app/start.sh'
  EXPOSE: "8080"
  WORKDIR: /app
shell:
  beforeInstall:
    - apt-get update || true
    - apt-get install -y -q --no-install-recommends curl build-essential libfontconfig1-dev
    - curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain stable -y
    - source $HOME/.cargo/env
    - curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash
    - export NVM_DIR="$HOME/.nvm"
    - '[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm'
    - nvm install --lts
    - npm install -g cargo-cp-artifact yarn
    - rm -rf /var/lib/apt/lists/*
  setup:
    - cd /app
    - export PATH="$HOME/.cargo/bin:$PATH"
    - cargo --version
    - export NVM_DIR="$HOME/.nvm"
    - '[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm'
    - nvm use --lts
    - node --version
    - yarn install --frozen-lockfile
    - (cd node_modules/skia-canvas && rm lib/v6/index.node && npm run install)
    - rm yarn.lock

    - echo 'export NVM_DIR="$HOME/.nvm" && [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" && nvm use --lts && node main.js' > start.sh

These are werf.io build instructions but it should be pretty easy to make a dockerfile from these. I just threw this together to work, it should be less complicated once prebuilt builds with gpu support are available.

The important parts here are:

• the from nvidia/vulkan:1.2.133-450 which adds the vulkan drivers to the container
• installing rust and node
• building skia-canvas

I'm running the image on a kubernetes cluster with gpu support but I think it should work with something like this as well: https://docs.docker.com/compose/gpu-support/

I'd been thinking of sticking with OpenGL since it stood the best chance of running on all three supported platforms (and using something like glutin for windowing).

I also first tried using OpenGL but I wasn't able to create a opengl context in docker, it was always missing an X11 or Wayland instance. With vulkan it works flawlessly though. It might be possible to pass through an X11 instance into the container from what I've read but that wouldn't really fit my usecase well. I was also not able to start an X11 context in the container itself.

But I agree, for rendering to a window on a desktop OS opengl is probably better supported than vulkan.

I should be able to create a example docker image next week and I'll also test if your PR works on windows and works on my kubernetes cluster!

[samizdatco]

I've made some progress on the rendering-to-a-window front, but haven't totally solved the problem and could greatly benefit from some suggestions at this point.

Here's a new repository of Windowing Prototypes where I've been attempting to go from Skia to the screen on all the platforms we need to support. You can run the tests in each of the subfolders by cd'ing into the metal, gl, or vulkan directory and typing cargo run.

So far, the only one that's fully functional is the Mac version using the Metal backend. I'd like to find a single solution that works for Linux & Windows (which effectively means either OpenGL or Vulkan), but have only gotten partway there with either of them.

I could really use some help debugging the two dead-ends I've reached:

1. Figuring out how to keep the ‘current’ GL context tied to the correct window (and its Skia context)
2. Fixing the validation errors in the Vulkan render pass that are preventing the graphics pipeline from completing

Both of these issues are fleshed out in more detail in the repo's Readme, along with pointers to the spots in the code where I think the problems (and solutions?) may lie...

[konsumer]

I can verify Mac/Metal demo runs great.

OpenGL runs great on my mac:

But like you said, window-resize or close makes the other windows weird. What if we just didn't allow window resize after init? What if we only did a single window?

If I change code to show just 1 window, it works great, resize is fine, and it seems ok to close it.

https://user-images.githubusercontent.com/83857/180625039-67b1d8e8-f8cf-4056-95dc-851d01d05808.mov

I am only testing on mac right now, but I can test on others, if you want.

[samizdatco]

One thing I forgot to mention about Vulkan: to use it on the Mac you'll first need to install the Vulkan SDK and set some environment variables before running the test. It'll look something like:

export VULKAN_SDK="/path/to/vulkan-sdk/macOS"
export DYLD_LIBRARY_PATH="$VULKAN_SDK/lib:$DYLD_LIBRARY_PATH"
export VK_ICD_FILENAMES="$VULKAN_SDK/etc/vulkan/icd.d/MoltenVK_icd.json"
export VK_LAYER_PATH="$VULKAN_SDK/etc/vulkan/explicit_layer.d"

I also found that I needed to create the etc directory myself and make a symlink to share/vulkan to satisfy the compiler:

❯ ls -l macOS/etc/vulkan
lrwxr-xr-x 1 cds admin 16 Jul 19 22:11 macOS/etc/vulkan -> ../share/vulkan/

[konsumer]

Ok, got vulkan SDK installed, and setup right (I think) and I did the symlink to etc. I get a black window and lots of output: https://gist.github.com/samizdatco/e0b37cd87dfbd4685555e3e39f9d9a2c

[samizdatco]

That looks pretty much identical to what I'm seeing on macOS as well. I'd be curious to see what it does on a Windows or Linux system with a GPU though if anyone can test that and post the output.

[samizdatco]

The Vulkan demo in the Windowing Prototypes repo now works for me on macOS (downgrading MoltenVK to a 1.2.x version did the trick) and I've gotten it to use a similar threading setup to the Metal demo. It'd help a ton if someone could try running the Vulkan demo on Windows and/or Linux and let me know if it does anything unexpected on either of those platforms.

[lucasmerlin]

I was finally able to test the gpu branch on windows and all seems good, the tests ran through. Also works with my docker / kubernetes setup, we're using it now on hellopaint.io!

Also tried the windowing repo on windows: With opengl it's flickering a lot, but is smooth as long as a resize, when I let go it starts flickering again. All the windows are flickering, even before I resize one window. https://user-images.githubusercontent.com/8009393/181111898-a2432c5c-52c7-4d73-b92f-87482326d782.mp4

With vulkan it's working smoothly: (cant video record multiple windows on windows unfortunately) Also I can note that I didn't have to install any sdk on windows like you have to on mac (but maybe some game already installed it for me previously) And I had to remove the platform::macos::WindowExtMacOS, import, which was not used anyways

[samizdatco]

I've reached the testing phase for this PR and would love to know how it works on other systems. The big change is the addition of a Window class which encapsulates a Canvas and provides event-handling hooks. Documentation for Window and the associated App global can be found in the branch's readme and other details are spelled out in the PR comment.

You can install the current release candidate (rc5) via:

npm install https://github.com/samizdatco/skia-canvas

And here's a simple demo that shows off mouse & keyboard events:

const {Window} = require('skia-canvas')

let win = new Window(400, 300),
    {canvas, ctx} = win // use the canvas & context created by the window

win.on('mousemove', ({button, x, y}) => {
  if (button == 0){ // a left click
    ctx.fillStyle = `rgb(${Math.floor(255 * Math.random())},0,0)`
    ctx.beginPath()
    ctx.arc(x, y, 10 + 30 * Math.random(), 0, 2 * Math.PI)
    ctx.fill()
  }

  win.cursor = button === 0 ? 'none' : 'crosshair'
})

win.on('keydown', ({key}) => {
  if (key == 'Escape'){
    ctx.clearRect(0, 0, canvas.width, canvas.height)
  }
})

[samizdatco]

The PR has landed in the 1.0 release. Closing this issue but please open new ones with any windowing-specific bugs that crop up.