Camera preview color conversions on GPU

[oscartbeaumont]

This PR is just AI slop but it seems to be working on my machine. It drops the ffmpeg rescaler in favor of using shaders for the color conversions. This should theorically be much faster and more efficient.

This needs an insane review and cleanup before we should even concider merging it.

TODO:

• [ ] Fix the black background on the camera preview

Summary by CodeRabbit

• New Features

  • GPU-accelerated camera preview with automatic CPU fallback for broader device support.
  • Frontend-accessible camera diagnostics: test, diagnose, quick-fix, auto-fix, and window status checks.
  • Optional native camera preview mode toggle.

• Improvements

  • Faster, smoother camera preview with better logging and health checks.
  • More reliable preview initialization and graceful shutdown behavior.

• Bug Fixes

  • Reduced cases of invisible or stuck camera preview/windows.
  • Stopping recordings now cleanly terminates the preview.

• Documentation

  • Added “Camera Preview Debug Guide” with workflows, examples, and troubleshooting tips.

[coderabbitai[bot]]

Walkthrough

Adds GPU-accelerated camera frame conversion via a new gpu-converters crate and integrates a native camera preview path with diagnostics, Tauri debug commands, and shutdown coordination. Updates window wiring to initialize the preview, expands logging, and adds documentation and benchmarking/tests for the GPU path.

Changes

Cohort / File(s)	Summary
Docs: Camera Debug Guide CAMERA_DEBUG_GUIDE.md	New guide for diagnosing invisible camera previews, with Rust/TS examples, logging tips, and health-check workflows.
Desktop: Camera preview core & diagnostics apps/desktop/src-tauri/src/camera.rs	Adds GPUCameraConverter usage with fallback, internal frame channel, cancellation token, expanded CameraPreview API, diagnostics helpers, window/feed tracking, and extensive logging.
Desktop: Tauri debug commands apps/desktop/src-tauri/src/commands/mod.rs, apps/desktop/src-tauri/src/commands/camera_debug.rs	New Tauri commands exposing camera test/diagnose/quick-fix/auto-fix and window status via CameraDebugReport.
Desktop: App integration apps/desktop/src-tauri/src/lib.rs, apps/desktop/src-tauri/src/recording.rs, apps/desktop/src-tauri/src/windows.rs	Wires native preview sender vs legacy path; initializes preview window; shifts cleanup to CameraPreview::shutdown; adjusts ShowCapWindow::Camera variant to struct-like; adds logs.
Desktop: Frontend touch-up apps/desktop/src/routes/camera.tsx	Removes unused variable from NativeCameraPreviewPage.
Manifests apps/desktop/src-tauri/Cargo.toml	Adds path dependency cap-gpu-converters to include the new crate.
GPU Converters: Crate setup crates/gpu-converters/Cargo.toml, crates/gpu-converters/README.md	New crate with features, dependencies (bytemuck, optional tokio), example benchmark, and comprehensive README.
GPU Converters: Public API and orchestration crates/gpu-converters/src/lib.rs, .../perf.rs, .../texture_pool.rs, .../fallback.rs	Introduces CameraFormat/CameraInput, GPUCameraConverter, ConversionPreset, performance tracking, texture pooling, and fallback strategies/errors.
GPU Converters: Format converters .../bgra_rgba/mod.rs, .../rgb24_rgba/mod.rs, .../nv12_rgba/mod.rs, .../uyvy_rgba/mod.rs, .../yuv420p_rgba/mod.rs, .../yuyv_rgba/mod.rs	GPU converters for multiple formats with async constructors, convert_to_texture and CPU readback paths; explicit device/queue usage and error handling.
GPU Converters: Shaders .../bgra_rgba/shader.wgsl, .../rgb24_rgba/shader.wgsl, .../nv12_rgba/shader.wgsl, .../uyvy_rgba/shader.wgsl, .../yuv420p_rgba/shader.wgsl, .../yuyv_rgba/shader.wgsl	WGSL compute shaders for format conversions and input type tweaks (e.g., float textures, storage buffers).
GPU Converters: Scaler .../scaler/mod.rs, .../scaler/nearest.wgsl, .../scaler/bilinear.wgsl, .../scaler/bicubic.wgsl	GPU scaling with nearest/bilinear/bicubic pipelines; quality selection and uniform-driven dimensions.
GPU Converters: Utilities, examples, tests .../examples/benchmark.rs, .../tests/integration_test.rs, .../util.rs	Benchmark harness, integration tests (mostly ignored), and minor util attribute.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant Camera as Camera Feed
  participant App as Tauri Backend
  participant Prev as CameraPreview
  participant GPU as GPUCameraConverter
  participant Fallback as CPU Fallback
  participant Win as Camera Window

  Camera->>Prev: RawCameraFrame (send via internal channel)
  Prev->>Prev: Receive frame
  alt GPU available
    Prev->>GPU: convert_and_scale(input, target, quality)
    GPU-->>Prev: RGBA bytes or error
    opt GPU error
      Prev->>GPU: analyze error
      Prev->>Fallback: convert_with_fallback(...)
      Fallback-->>Prev: RGBA bytes or error
    end
  else GPU unavailable
    Prev->>Fallback: CPU conversion/scale
    Fallback-->>Prev: RGBA bytes
  end
  Prev->>Win: upload/render RGBA
  Note over Prev,Win: Logs counters, texture uploads, render passes

sequenceDiagram
  autonumber
  participant FE as Frontend (TS)
  participant Cmd as Tauri Commands
  participant Prev as CameraPreview
  participant Diag as CameraDiagnostics
  participant Win as Camera Window

  FE->>Cmd: diagnose_camera_preview(window)
  Cmd->>Diag: diagnose_camera_preview(&Prev, &Win)
  Diag-->>Cmd: Diagnostic report (String)
  Cmd-->>FE: CameraDebugReport { success, details }

  FE->>Cmd: quick_fix_camera_preview(window)
  Cmd->>Diag: quick_fix_camera_preview(&Prev, &Win)
  Diag-->>Cmd: Vec<String> fixes
  Cmd-->>FE: CameraDebugReport { fixes_applied }

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Possibly related PRs

• CapSoftware/Cap#895 — Adjusts native camera preview initialization in src-tauri; overlaps with init_preview_window wiring and startup handling.
• CapSoftware/Cap#945 — Modifies camera preview APIs and lifecycle/shutdown; overlaps with CameraPreview changes and window management.

Suggested reviewers

• Brendonovich

Poem

A rabbit boots the GPU with gleam,
Swizzles YUV into RGBA stream.
If shaders stumble, CPU hops in,
Windows awake, preview begins.
Commands diagnose, fixes align—
Frames now dance, perfectly fine.
(/) () 🐇 pixels divine!

[!TIP]

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

✨ Finishing Touches

• [ ] 📝 Generate Docstrings

🧪 Generate unit tests

• [ ] Create PR with unit tests
• [ ] Post copyable unit tests in a comment
• [ ] Commit unit tests in branch camera-preview-gpu-accel

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[oscartbeaumont]

@coderabbitai review

[coderabbitai[bot]]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.