Use library for screen input

[gen2brain]

There is a library here https://github.com/kbinani/screenshot with native (no cgo) support for X11 and Windows, and with macOS cgo support. Instead of implementing for every OS perhaps it is better to just use the already existing library and improve it further if there is a need. Currently, your X11 implementation uses cgo, and one also must install dev/devel headers on most distros, my suggested implementation uses BurntSushi native library for X11, and some time ago I also added a native shm support in kbinani/screenshot.

Btw. malgo context now have DeviceInfo() if you need to use it. I also helped with static tag in go-sdl2 project, where there are included static libraries for common platforms, https://github.com/veandco/go-sdl2/blob/master/sdl/sdl_cgo_static.go , and with libraries here https://github.com/veandco/go-sdl2/tree/master/.go-sdl2-libs so you can check it, for example, it would be very nice if the user doesn't have to worry about dependencies.

[lherman-cs]

@gen2brain https://github.com/kbinani/screenshot looks very interesting! I will definitely try that. It seems to be a great option for mediadevices.

Btw. malgo context now have DeviceInfo() if you need to use it. I also helped with static tag in go-sdl2 project, where there are included static libraries for common platforms, https://github.com/veandco/go-sdl2/blob/master/sdl/sdl_cgo_static.go , and with libraries here https://github.com/veandco/go-sdl2/tree/master/.go-sdl2-libs so you can check it, for example, it would be very nice if the user doesn't have to worry about dependencies.

Thanks for all of these info!! They're really helpful. I like how the static links are organized here, https://github.com/veandco/go-sdl2/blob/master/sdl/sdl_cgo_static.go.

[lherman-cs]

@gen2brain I've added https://github.com/kbinani/screenshot for display capture, and your DeviceInfo API. Thanks!!

By the way, how did you generate a big platform permutation of static libraries, https://github.com/veandco/go-sdl2/blob/master/sdl/sdl_cgo_static.go? I checked the travis and Github action configs, but I couldn't see them.

[gen2brain]

@lherman-cs In my distro, it is easy to compile toolchain/compiler for whatever platform/OS (except macOS, but there is https://github.com/tpoechtrager/osxcross for that), for the example see here https://github.com/gen2brain/cam2ip/blob/master/make.bash#L3.

So far I did the libraries update when needed, but I know the maintainer of go-sdl2 now has some solution that uses Github actions, but probably not yet used. Maybe you can contact him directly and ask about that.

[gen2brain]

Btw. there is another method I prefer to use, where possible, and that is to include all .c source files needed and that will build one object file. Much nicer than including static libraries, but it cannot be used everywhere. See for example https://github.com/gen2brain/aac-go/blob/master/aacenc/aacenc_cgo.go and https://github.com/gen2brain/x264-go/blob/master/x264c/x264c_cgo.go. I know glfw bindings also use that approach.

[at-wat]

Is there a performance difference between cgo version and kbinani/screenshot? If there is, the old one can be remained as a separate driver. (I also can move the old one into my product tree, there aren't a big problem.)

[at-wat]

I roughly compared the performance using rtp example with FullHD screen on my laptop. The commit 356eee1 (cgo version), uses 90% of single thread. Current master commit c734c53b0060bf564482a17d69dc0e315a323978 (kbinani/screenshot version) uses 135%.

[lherman-cs]

@at-wat During my test, I didn't see a significant latency difference. But, kbinani/screenshot definitely uses more CPU. I think we can put in the same package, and uses conditional build to use the previous version for Linux. This way, we can get the best of both worlds.

[lherman-cs]

@at-wat create a PR to add the original x11 adapter back, https://github.com/pion/mediadevices/pull/258.

I'm curious to see why the CPU usage difference is really big. Is that only because cgo vs go? It would be really awesome if we can try to optimize the non-cgo version so that we can have a more common codebase for multiple platforms.

[lherman-cs]

@gen2brain Oh that makes sense. That's definitely a nice approach, this means we only need 1 OS and simply install the right toolchains.

[rivercheng]

Currently, in my laptop (macbook pro 2019), screen capture takes more than 180ms. By checking the source code of kbinani/screenshot, I suspect the most time consuming part is to convert ARGB to RGBA. 'screenshot_darwin.go' 108 for iy := 0; iy < height; iy++ { 109 j := i 110 for ix := 0; ix < width; ix++ { 111 // ARGB => RGBA, and set A to 255 112 img.Pix[j], img.Pix[j+1], img.Pix[j+2], img.Pix[j+3] = img.Pix[j+1], img.Pix[j+2], img.Pix[j+3], 255 113 j += 4 114 } 115 i += img.Stride 116 } This conversion is not necessary if we later encode these frames to VP8 or H264, so it is better to create a customized image.Image type to support ARGB and avoid the conversion.

Even if we really want to convert it to RGBA (e.g for PNG), we'd better to provide a separate function and implement it in an optimized C.

[rivercheng]

Sorry, my suspicion above is wrong. It is not mainly because of this re-order. I think the API itself is slow.

Maybe we should try to use CaptureScreenInput. Here's an example: https://github.com/tomashanacek/screen-recorder. Or we can use CGDisplayStream, e.g. libwebrtc and https://github.com/cretz/go-scrap.