Simulator camera

[AvdLee]

trafficstars

Create an option to Simulator the camera. In case the camera would be requested, we can launch RocketSim camera mode, create an image using the Mac camera, and return it as a file within the Simulator.

[im-jersh]

Would love this feature. I have image capture and barcode scanning components in my app's flows and reaching for a physical device to work on these features really slows me down, especially if I'm working away from my desk

[zigfreid5]

This would be great! One of those things that you wonder why Apple hasn't done already.

[AvdLee]

Making progress on this feature: https://x.com/twannl/status/1926715711277203563

Now testing with actual projects.

• [x] Looki (CoreML scanning app)
• [x] AVCam (Apple's example project)
• [ ] Apple's Scanning App
• [ ] Hidde's app
• [ ] Jordi's app
• [ ] React Native Vision Camera
• [x] Glimpsify
• [ ] MovingBox (See reply)

If you're reading this: I'm looking for open-source camera projects to test this feature thoroughly!

Useful resources

• https://nshint.io/blog/2019/04/08/testing-the-camera-on-the-simulator/

Todo

• [x] Create a xcworkspace for easy testing
• [x] Create a macOS and iOS SPM package
• [x] Integrate the iOS package into RocketSimConnect so it works with existing code
• [x] Rewrite the code to make it work via the existing connection?
• [x] Integrate camera into RocketSim
• [x] Add proper logging using OSLog
• [x] Prevent main thread blocking when the camera is active
• [x] Check memory usage in-app when using RocketSim Connect
• [x] Reduce main thread usage in NWConnection handlers
• [x] Send camera start stop messages
• [ ] Show disconnected message in preview layer if so
• [x] Reduce frame rate for better performance
• [x] Warm up device
• [x] Check memory usage in Mac app
• [x] Create RocketSim Connect onboarding flow for Simulator Camera
• [x] Make the camera connect if streamer appears later (poll until connection becomes available)
• [x] Allow to disable/enable RocketSimConnect features from RocketSim's settings
• [ ] Add support for audio inside videos
• [x] Add capturing view to UIImagePickerController
• [ ] Add support for AVCaptureMetadataOutput

Camera Capturer

• It should only do work when the camera is actually required
• It should always listen for Simulator Camera requests

[camdenwebster]

Feel free to give this a test with MovingBox!

[AvdLee]

Feedback

Let's set up tests to cover all these incoming ones!

• [ ] videoZoomFactor

• [ ] Add RocketSim Connect setup button to preferences
• [ ] Add CMD + , to open preferences

[AvdLee]

Closing this one as the foundation is implemented. Continuing development in dedicated issues!