Camera / webcam Support

[mitchmindtree]

This is an issue for tracking camera/webcam support!

---

Desired Features

• Enumerate available camera devices on the system.
• Enumerate device info and capabilities.
  • Info should include a unique identifier / human-readable name.
  • Capabilities should include supported resolution, rate and format combos.
• Request permission to start a capture stream on a device with a desired stream configuration. Should return some async result representing user accepting permissions and stream starting.
• Receive captured frames via either a callback or iterator.
• Captured frame should provide size, format and pointer to data. We can create an easy/efficient API for loading this data into a wgpu::Texture.

Nice-to-haves

• Play/pause API. This could be worked around by destroying the stream and creating a new one.
• Pure-Rust down to the platform API.
• Callback/event notification of addition/removal of devices.

---

Existing Work

• uvc - Bindings libuvc: a cross-platform C library for USB video devices.
• camera_capture - linux, windows - A very basic lib for starting camera streams and displaying frames. Uses rscam for linux and escapi for windows.
• rscam - linux - a fairly thorough interface to the v4l (video4linux) library.
• escapi-rs - windows - bindings to the "extremely simple capture API" C++ library.
• webcam_stream_wasm - A wasm webcam stream WIP.
• "webcam" search on crates.io.

Unfortunately a lot of these boil down to using C/C++ bindings under the hood. rscam looks nice in that at least v4l seems to be the way to get access to cameras on linux.

Perhaps it's worth starting a project similar to CPAL, but for camera input rather than audio input/output? CPCL - cross-platform camera library?

[mulimoen]

Hi, I am the author of the libuvc bindings. I believe libuvc fulfills all the desired features, apart from being implemented in c, and not having event notifcation (this could be implemented through libusb). The bindings should be in a usable state right now, although still somewhat incomplete.

Is there anything this community would like from the libuvc bindings, or some wanted features? I would be happy to discuss how our libraries may be developed to meet the need of nannou and users

[Niko-La]

Hi, I am the author of the libuvc bindings. I believe libuvc fulfills all the desired features, apart from being implemented in c, and not having event notifcation (this could be implemented through libusb). The bindings should be in a usable state right now, although still somewhat incomplete.

Is there anything this community would like from the libuvc bindings, or some wanted features? I would be happy to discuss how our libraries may be developed to meet the need of nannou and users

Can we access a uvc webcam on mobile using otg cable through wasm?

Webrtc does not support this yet. The goal is to build pwa with usb webcam as input without the need to make native android app.

[mulimoen]

I am not sure if libusb would function on wasm, given that runtimes would need to passthrough the usb devices. webusb never really got implemented other than in Chrome, but maybe one could use emscripten here.

[mofeing]

I found nokhwa, that seems to tick some of the desired features.

The library basically unifies multiple backends (Video4Linux, libuvc, OpenCV, GStreamer, ...) into a common API. MSMF (Windows) & WASM support is experimental and AVFoundation (macOS) is planned/WIP.

• Enumerate available camera devices on the system.

query_devices function.

• Enumerate device info and capabilities.
  • Info should include a unique identifier / human-readable name.
  • Capabilities should include supported resolution, rate and format combos.

CameraInfo returned by query_devices contains human-readable name. Capabilities like resolution, rate and format are known once you create a Camera with resolution, frame_rate and frame_format methods.

• Captured frame should provide size, format and pointer to data. We can create an easy/efficient API for loading this data into a wgpu::Texture.

Can directly copy a frame to a wgpu texture with Camera.frame_texture method.

• Play/pause API. This could be worked around by destroying the stream and creating a new one.

Camera.open_stream, Camera.stop_stream 🙂

The only features that do not seem to be available are the async results and frame capture via callback or iterator (frame capture must be explicitly called via Camera.frame method).

[l1npengtul]

Hello, I am the maintainer of nokhwa.

I am interested in helping, feel free to open a issue over on nokhwa's issue board. For now, I will implement a method that allows a direct capture into a nannou wgpu texture.

Hopefully the new features required can be merged in by either 0.4 or 0.5.

E: I have also taken over maintainership of libuvc-rs. I plan to update libuvc after the 0.4 release of nokhwa (The libuvc backend is currently due to some compilation errors associated with it).

[l1npengtul]

There might be one issue - nokhwa's licensing is MPL-2.0, while nannou's licensing is MIT/APL2. If there is a need to change the license, let me know.

[l1npengtul]

I've updated nokhwa to 0.4 - it checks all the tickboxes now!

[patrick91]

@l1npengtul do you have an example of using nokhwa with nannou? 😊

[l1npengtul]

@l1npengtul do you have an example of using nokhwa with nannou? 😊

Sorry, I was busy (and assumed this was dead) so not yet, but I'll work on it!

[vvzen]

Hi all! any progress on this? this functionality would be quite cool to have in nannou! Thanks!

V