[camera] add video stabilization

[ruicraveiro]

trafficstars

Adds support for video stabilization to camera_platform_interface, camera_avfoundation, camera_android_camerax and camera packages.

The video stabilization modes are defined in the new VideoStabilizationMode enum defined in camera_platform_interface:

/// The possible video stabilization modes that can be capturing video.
enum VideoStabilizationMode {
  /// Video stabilization is disabled.
  off,

  /// Basic video stabilization is enabled.
  /// Maps to CONTROL_VIDEO_STABILIZATION_MODE_ON on Android
  /// and throws CameraException on iOS.
  on,

  /// Standard video stabilization is enabled.
  /// Maps to CONTROL_VIDEO_STABILIZATION_MODE_PREVIEW_STABILIZATION on Android
  /// (camera_android_camerax) and to AVCaptureVideoStabilizationModeStandard
  /// on iOS.
  standard,

  /// Cinematic video stabilization is enabled.
  /// Maps to CONTROL_VIDEO_STABILIZATION_MODE_PREVIEW_STABILIZATION on Android
  /// (camera_android_camerax) and to AVCaptureVideoStabilizationModeCinematic
  /// on iOS.
  cinematic,

  /// Extended cinematic video stabilization is enabled.
  /// Maps to AVCaptureVideoStabilizationModeCinematicExtended on iOS and
  /// throws CameraException on Android.
  cinematicExtended,
}

There is some subjectivity on the way with which I mapped the modes to both platforms, and here's a document that compares the several modes: https://docs.google.com/spreadsheets/d/1TLOLZHR5AcyPlr-y75aN-DbR0ssZLJjpV_OAJkRC1FI/edit?usp=sharing, which you can comment on.

List which issues are fixed by this PR. You must list at least one issue. Partially implements https://github.com/flutter/flutter/issues/89525

Pre-launch Checklist

• [x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
• [x] I read the Tree Hygiene page, which explains my responsibilities.
• [x] I read and followed the relevant style guides and ran the auto-formatter. (Unlike the flutter/flutter repo, the flutter/packages repo does use dart format.)
• [x] I signed the CLA.
• [x] The title of the PR starts with the name of the package surrounded by square brackets, e.g. [shared_preferences]
• [x] I linked to at least one issue that this PR fixes in the description above.
• [x] I updated pubspec.yaml with an appropriate new version according to the pub versioning philosophy, or this PR is exempt from version changes.
• [x] I updated CHANGELOG.md to add a description of the change, following repository CHANGELOG style, or this PR is exempt from CHANGELOG changes.
• [x] I updated/added relevant documentation (doc comments with ///).
• [x] I added new tests to check the change I am making, or this PR is test-exempt.
• [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

[google-cla[bot]]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[ruicraveiro]

Hi @hellohuanlin,

I have improved camera_avfoundation's implementation based on your comments. Instead of adding a new commit, I forced pushed a squash commit of those changes and another commit on top of it that adds the dependency_overrides to the pubspecs.

Assuming there will be more improvments I need to make, is it OK I keep adding commits until everything is fixed, and only then do a final squash merge, excluding the dependency_overrides commit, and force that squash commit?

Thanks!

[hellohuanlin]

Assuming there will be more improvments I need to make, is it OK I keep adding commits until everything is fixed, and only then do a final squash merge, excluding the dependency_overrides commit, and force that squash commit?

All commits will be auto squashed when landed.

[hellohuanlin]

LGTM on iOS. But please get stamps from other platforms too.

[ruicraveiro]

LGTM on iOS. But please get stamps from other platforms too.

Thanks!!

[camsim99]

Hi @ruicraveiro :) I don't see the Android changes that you mentioned. Is it possible you didn't push them?

[ruicraveiro]

Hi @camsim99 :-) Oh dear, I forgot to push them to GH... oopsie.

[ruicraveiro]

Thanks @camsim99!!!!

[ruicraveiro]

Just submitted https://github.com/flutter/packages/pull/7548 for only the platform interface.

[ruicraveiro]

Hi reviewers, after feedback from @stuartmorgan, I am resubmitting this with a revised set of enum options. They were made more abstract. So, now VideoStabilizationMode no longer has off, on, standard, cinematic and cinematicExtended, it has off, level1, level2 and level3. Android only maps to off and level1. iOS maps to all levels.

[stuartmorgan-g]

From triage: @ruicraveiro is this still something you are planning to come back to?

[ruicraveiro]

From triage: @ruicraveiro is this still something you are planning to come back to?

I would prefer to get it approved as it is, but yes my intention is to eventually complete the changes we discussed. I just don't know exactly when I will be able to pick this up again. Eventually, still this month, but if I'm unable, my next opportunity will be only from mid-2025 forward.

[stuartmorgan-g]

I would prefer to get it approved as it is

The current version conflates Exceptions and Errors as discussed in the long comment thread above, so it would not be approved as is; the version without fallback would still require changes.

[ruicraveiro]

I would prefer to get it approved as it is

The current version conflates Exceptions and Errors as discussed in the long comment thread above, so it would not be approved as is; the version without fallback would still require changes.

Sure. It will be easier for me to find the time if the required changes are limited to fixing the confusion between exceptions and errors, which will increase the odds of being able to do so this year instead of later in 2025.

[ruicraveiro]

Hi @stuartmorgan,

I think next week I'll find enough time to finish this off. But, before doing so, I want to get a full agreement on the specification of setTargetVideoStabilizationMode, so I can make the best use of my time.

Error Handling

First, concerning error handling:

• For platforms without an implementation, we throw UnimplementedError.
• For platforms with implementation, but without any capability, right now I am returning an empty array. I am not sure if this is the ideal behaviour, or if we should always return at least [ VideoStabilizationMode.off ]. What do you think? I will assume we'll keep the current implementation for the rest of this comment.
• Methods already implemented in CameraController check to see if the camera is already initialized before doing their thing by calling _throwIfNotInitialized, which throws a CameraException if they are called before initialization. While I would argue that this is also a programming error, so it would make sense to throw an error instead of an exception, I see a stronger reason to follow the pattern and call _throwIfNotInitialized just like every other method is doing. This is what is currently implemented.

Fallback behaviour

Concerning the implementation with a fallback, here's a table of what setTargetVideoStabilizationMode should do:

requested mode	no supported mode ([])	only off supported	max level1 supported	max level2 supported	max level3 supported
off	throw ArgumentError()	off	off	off	off
level1	throw ArgumentError()	throw ArgumentError()	level1	level1	level1
level2	throw ArgumentError()	throw ArgumentError()	level1	level2	level2
level3	throw ArgumentError()	throw ArgumentError()	level1	level2	level 3

The implementation of this logic would be in the camera package, not in the platform implementations to ensure it is always consistent. I don't see a reason to place it in the platform implementation packages.

Extra

Because of the time distance between the last time we discussed about this and now, I had to go over the entire discussion from scratch and while doing so it struck me that we could have it both ways. You wanted to make it easier for developers that want to just have the best available mode up to X and I wanted a more strict approach to cater to developers who, like me, prefer less nuanced behaviour. So, it occurred to me that the default behaviour could be the fallback behaviour, but we could add an optional bool named argument, for example, strict, which would revert the method to, as it says, a strict behaviour. So, the method's final signature would be Future<VideoStabilizationMode> setTargetVideoStabilizationMode(VideoStabilizationMode mode, {bool strict = false}), and when strict would be true, the behaviour would be:

requested mode	no supported mode ([])	only off supported	max level1 supported	max level2 supported	max level3 supported
off	throw ArgumentError()	off	off	off	off
level1	throw ArgumentError()	throw ArgumentError()	level1	level1	level1
level2	throw ArgumentError()	throw ArgumentError()	throw ArgumentError()	level2	level2
level3	throw ArgumentError()	throw ArgumentError()	throw ArgumentError()	throw ArgumentError()	level3

I don't want to spend much more energy discussing this, so if you don't agree with this extra (strict), it's OK and I'll just implement without it, but if you do, I think we would be striking a really good balance here. The only reason I see against this is that it will require an effort from me to document this in such a way that it is very easy and clear to understand.

Let me know what you think.

Thanks!

Rui

[ruicraveiro]

iOS part LGTM

Thanks!!

[stuartmorgan-g]

I think next week I'll find enough time to finish this off.

Sorry I was not able to respond before then; I was out of office for several weeks starting shortly before that comment.

• For platforms with implementation, but without any capability, right now I am returning an empty array. I am not sure if this is the ideal behaviour, or if we should always return at least [ VideoStabilizationMode.off ]. What do you think?

I agree with empty array; I think not including off in the set of things that method can return will make it much easier for clients to use.

So, it occurred to me that the default behaviour could be the fallback behaviour, but we could add an optional bool named argument, for example, strict, which would revert the method to, as it says, a strict behaviour. [...] The only reason I see against this is that it will require an effort from me to document this in such a way that it is very easy and clear to understand.

That sounds like a reasonable API, so adding it to better support your use case sounds good to me. I don't think it should be too hard to explain in the doc comment what it means; my suggestion would be to reverse its meaning and make it allowFallback = true, since I think that's more self-explanatory in code, and will thus be easier to document.

[gmr-renato]

Hi guys, any progress on this one? I'm working on a sport's recording camera and video stabilization is a huge feature. @ruicraveiro can I help you to progress on this?

[ruicraveiro]

Hi guys, any progress on this one? I'm working on a sport's recording camera and video stabilization is a huge feature. @ruicraveiro can I help you to progress on this?

Hi Renato,

No, not at this point. Maybe during the next couple of weeks I'll have time to finish this off.

Rui

[ruicraveiro]

Hi @stuartmorgan, I have just pushed an update to this branch with the following changes:

• At the platform levels, camera_android_camerax and camera_avfoundation, if setVideoStabilizationMode() is called with a mode that is unavailable, then that method with throw an ArgumentError. I also figured a cleaner way to map between VideoStabilizationMode and the platform's specific representation of the mode. Now I am mapping both ways using a single method, which in both platforms is called _getSupportedVideoStabilizationModeMap(). I also fixed a couple of nits picked up by @hellohuanlin.

• Because it is a common feature with a common logic, I implemented the fallback logic in CameraController, in the camera package, for setVideoStabilizationMode() method, as specified in my message on the 20th December. When allowFallback is true, then CameraController will only call the platform's implementation with a supported video stabilization mode and will throw ArgumentException if none is available (or if more than off is asked for and only off is available). When allowFallback is false, then CameraController passes the mode as is to the platform's implementation.

I have some broken tests, but I don't think they're related to anything I did, except merging the main branch onto this one. On my machine, all Dart unit tests related to this package are OK. Native tests are indeed failing, but they're also failing when I run them on the latest version of the main branch (plus I'm getting failed Dart tests for other packages on both this branch and main). Except for the very tiny nits (bool -> BOOL and a couple of empty lines removed), all changes were made to Dart code, not native code.

[FabioCFonseca]

@ruicraveiro Great work! I'm closely monitoring this PR

[leooruiz]

Nice work @ruicraveiro! I'm keeping an eye on this and plan to test it.

[MarquiThiago]

I'm planning to test this implementation in real-case scenarios with multiple devices by the time you're done, keep up with the amazing work @ruicraveiro!

[leooruiz]

Hello @ruicraveiro,

Thank you for your work on this stabilization PR. I tested the update on my sports recording app and encountered a compilation error on iOS. Below is the error message from the console.

Error log:

Launching lib/main.dart on iPhone 11 in release mode...
Automatically signing iOS for device deployment using specified development team in Xcode project: H9UAZA7PV4
Xcode build done.                                           59.5s
Failed to build iOS app
Could not build the precompiled application for the device.
ARC Semantic Issue (Xcode): No visible @interface for 'NSObject<FLTCaptureDeviceFormat>' declares the selector 'isVideoStabilizationModeSupported:'
/Users/leo_ruiz/repositories/xports/flutter_packages/packages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m:1246:36

ARC Semantic Issue (Xcode): No visible @interface for 'NSObject<FLTCaptureDeviceFormat>' declares the selector 'isVideoStabilizationModeSupported:'
/Users/leo_ruiz/repositories/xports/flutter_packages/packages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m:1262:38
2

Error running application on iPhone 11.

Exited (1).

Before compiling, I ran flutter clean and flutter pub get.

Environment details:

• Flutter: 3.27.1 (stable channel)
• Dart: 3.6.0
• macOS: Version 15.2 (M1 Pro)
• Xcode: 16.2
• Device: iPhone 11 running iOS 18.3.2

Flutter doctor output:

❯ flutter doctor -v
[✓] Flutter (Channel stable, 3.27.1, on macOS 15.2 24C101 darwin-arm64, locale en-BR)
    • Flutter version 3.27.1 on channel stable at /Users/leo_ruiz/.asdf/installs/flutter/3.27.1
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 17025dd882 (3 months ago), 2024-12-17 03:23:09 +0900
    • Engine revision cb4b5fff73
    • Dart version 3.6.0
    • DevTools version 2.40.2

[✓] Android toolchain - develop for Android devices (Android SDK version 35.0.0)
    • Android SDK at /Users/leo_ruiz/Library/Android/sdk
    • Platform android-35, build-tools 35.0.0
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 16.2)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 16C5032a
    • CocoaPods version 1.16.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2022.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)

[✓] VS Code (version 1.98.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.106.0

[✓] Connected device (5 available)
    • SM A055M (mobile)               • R9XX60289MX               • android-arm64  • Android 14 (API 34)
    • iPhone 11 (mobile)     • 00008030-0014059A1ED0402E • ios            • iOS 18.3.2 22D82
    • macOS (desktop)                 • macos                     • darwin-arm64   • macOS 15.2 24C101 darwin-arm64
    • Mac Designed for iPad (desktop) • mac-designed-for-ipad     • darwin         • macOS 15.2 24C101 darwin-arm64
    • Chrome (web)                    • chrome                    • web-javascript • Google Chrome 134.0.6998.166
    ! Error: Browsing on the local area network for iPad. Ensure the device is unlocked and attached with a cable or associated with the same local area network as this Mac.
      The device must be opted into Developer Mode to connect wirelessly. (code -27)

[✓] Network resources
    • All expected network resources are available.

• No issues found!

If you need any further information or additional testing on my side, please let me know. Thanks again for your support and the excellent work on this PR!

[stuartmorgan-g]

I have some broken tests, but I don't think they're related to anything I did, except merging the main branch onto this one. On my machine, all Dart unit tests related to this package are OK. Native tests are indeed failing, but they're also failing when I run them on the latest version of the main branch

I'm not sure why you would be getting failures on main; we run the tests continuously in CI. But the compile failures in the native tests here are related to the changes in the PR; likely the PR just needs to be updated to add the methods you are calling to the wrappers that were recently added as part of refactoring of the camera implementation to increase testability and allow for a migration to Swift (where we can't use OCMock).

[FabioCFonseca]

Hi @ruicraveiro,

I tested this PR on Android, and it worked fine; the crop indicates stabilization is being applied.

I’m just unsure about the differences among VideoStabilizationMode levels 1, 2, and 3, as I couldn’t visually distinguish any variations...

Enviroment:

Device: Motorola Edge 20 - Android 13

Doctor summary (to see all details, run flutter doctor -v): [✓] Flutter (Channel stable, 3.27.1, on macOS 14.4.1 23E224 darwin-arm64, locale en-BR) [✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0) [✓] Xcode - develop for iOS and macOS (Xcode 15.4) [✓] Chrome - develop for the web [✓] Android Studio (version 2023.3) [✓] VS Code (version 1.97.0) [✓] Connected device (4 available) [✓] Network resources

[stuartmorgan-g]

I tested this PR on Android, and it worked fine; the crop indicates stabilization is being applied.

I’m just unsure about the differences among VideoStabilizationMode levels 1, 2, and 3, as I couldn’t visually distinguish any variations...

Android only supports level 1.

[romainfd]

Hi, thank you and congratulations @ruicraveiro (and @stuartmorgan-g) for the amazing work on this! I am very interested in this feature, is there any way for me to help you progress on it?

[bparrishMines]

@ruicraveiro It looks like the camera_android_camerax ProxyApi update broke your changes to that plugin. If you are still working on this, let me know if you need help updating that portion or I can also quickly update the API wrapper for you.

[gyenesvi]

Hi @ruicraveiro, is this still ongoing work or is it stalled? Any ideas how much work is required / when this could become available in Flutter? Would be nice to understand if this feature is worth waiting for, as video recording in its current form is not very useful for my purposes. Thanks!