[camera_avfoundation] handle interruptions and use single offset

[misos1]

Handle interruptions of capture sessions by doing "internal pause". Another possibility would be to remove audio input from the session but this would be too specific for interruptions caused by an incoming call or alarm. Or to stop video recording as the native ios camera app does but this would require some additional event.

Handle asynchronous error notifications posted about capture sessions. This can happen for example when there is an incoming phone call and the camera starts recording with audio. For this specific case would be enough to just call startRunning on capture sessions but that is too specific and would require ability to restart capture session from dart side as seems there is no good way how to test when such phone call or other such event ends, and trying to restart it for example every second inside FLTCam does not look like good idea. Instead the camera controller enters an error state and can be recovered by disposing and recreating which should be more generic.

Use single offset for both video and audio as separate offsets can cause unsynchronized video with audio. Seems audio timestamps tend to be ignored except for the first sample so prefer them for calculating offset to avoid desynchronization. Avoid errors when trying to append samples with timestamp less or equal to the previous sample. Fix a bug when after adjusting offset also last sample time needs to be updated to avoid adding past offsets multiple times.

• fixes https://github.com/flutter/flutter/issues/149978
• fixes https://github.com/flutter/flutter/issues/151253

Pre-Review Checklist

• [x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
• [x] I read the Tree Hygiene page, which explains my responsibilities.
• [x] I read and followed the relevant style guides and ran the auto-formatter.
• [x] I signed the CLA.
• [x] The title of the PR starts with the name of the package surrounded by square brackets, e.g. [shared_preferences]
• [x] I linked to at least one issue that this PR fixes in the description above.
• [x] I updated pubspec.yaml with an appropriate new version according to the pub versioning philosophy, or I have commented below to indicate which version change exemption this PR falls under[^1].
• [x] I updated CHANGELOG.md to add a description of the change, following repository CHANGELOG style, or I have commented below to indicate which CHANGELOG exemption this PR falls under[^1].
• [x] I updated/added any relevant documentation (doc comments with ///).
• [x] I added new tests to check the change I am making, or I have commented below to indicate which test exemption this PR falls under[^1].
• [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

[^1]: Regular contributors who have demonstrated familiarity with the repository guidelines only need to comment if the PR is not auto-exempted by repo tooling.

[misos1]

Mac_arm64 ios_platform_tests_shard_2 master
Testing failed:
	Type 'AVCaptureSession' has no member 'wasInterruptedNotification'
	Command SwiftCompile failed with a nonzero exit code
	Testing cancelled because the build failed.

I am not sure what this should mean. AVCaptureSession.wasInterruptedNotification should be there since ios 4 (I managed to run tests successfully locally on real device and also on simulator).

[stuartmorgan-g]

From triage: what's the status of this PR? It's not clear to me if it's waiting for updates, or waiting for re-review.

[misos1]

It is waiting for updates.

[misos1]

Test testImageStreamEventFormat is failing locally with XCTAssertEqual(camera.isStreamingImages, true) XCTAssertEqual failed: ("false") is not equal to ("true"). Seems waitForQueueRoundTrip is "waiting" for the wrong queue as isStreamingImages is changed on captureSessionQueue. But even with the right queue it cannot wait on something which was not already dispatched at the time waitForQueueRoundTrip was called, and the block where that variable changes may not be. There should be a better way to wait for some variable to change in xc test.

[stuartmorgan-g]

From triage: Ping @hellohuanlin on this review.

[hellohuanlin]

@misos1 I took another look at the PR, and still feel hard to fully digest the change. It sounds like you are trying to do multiple things at once:

1. handling phone call interrupts where we can pause the recording
2. handling system errors that's not recoverable
3. fixing audio/video sync

I'm wondering if we can split it into separate PRs so it's easier to review.

[stuartmorgan-g]

From triage: @misos1 Are you planning on updating this PR based on the feedback above?

[misos1]

@hellohuanlin Points 1. and 2. are just different instantiations of the same problem. It has just different manifestations. In one case it sends interruption and in another case it sends an error. Point 3. is an essential part of the fix. Without it would be useless to disconnect as it would not work like pause for both audio and video so audio would be shifted.

[okorohelijah]

@hellohuanlin review from ios-triage

[hellohuanlin]

@misos1 To make review easier, could you update your PR description? I have some trouble following it. Please also explain what the bug is.

Handle interruptions of capture sessions by doing "internal pause".

What is "internal pause"

Another possibility would be to remove audio input from the session but this would be too specific for interruptions caused by an incoming call or alarm.

Why do we need to remove audio input? When to do it? why is it too specific?

Or to stop video recording as the native ios camera app does but this would require some additional event.

what additional event?

Handle asynchronous error notifications posted about capture sessions. This can happen for example when there is an incoming phone call and the camera starts recording with audio.

start recording before or after incoming phone call?

For this specific case would be enough to just call startRunning on capture sessions but that is too specific and would require ability to restart capture session from dart side as seems there is no good way how to test when such phone call or other such event ends, and trying to restart it for example every second inside FLTCam does not look like good idea.

Could you split this long sentence so it's easier to understand? startRunning on audio or video? Why requires ability to restart form dart side?

Instead the camera controller enters an error state and can be recovered by disposing and recreating which should be more generic.

Is this what you plan to do or what the current state is?

Use single offset for both video and audio as separate offsets can cause unsynchronized video with audio.

Is this what you plan to do or what the current state is?

Seems audio timestamps tend to be ignored except for the first sample so prefer them for calculating offset to avoid desynchronization.

Who ignores the audio timestamps? flutter or apple? What does "them" refer to? the "audio samples"? or the "first sample"?

Avoid errors when trying to append samples with timestamp less or equal to the previous sample.

How to avoid?

Fix a bug when after adjusting offset also last sample time needs to be updated to avoid adding past offsets multiple times.

What bug is fixed?

[LouiseHsu]

@misos1 Do you think you will be able to make the requested changes? :)

[misos1]

@hellohuanlin

Handle interruptions of capture sessions by doing "internal pause".

What is "internal pause"

That meant to not process any video and audio samples like when the camera controller paused.

Another possibility would be to remove audio input from the session but this would be too specific for interruptions caused by an incoming call or alarm.

Why do we need to remove audio input? When to do it? why is it too specific?

Removing audio input during a call would enable the camera to continue without audio. But this is specific to calls and alarms. For example what if interruption is due to something that wants exclusive access to the device camera? I did not see any simple way to detect whether it will be enough to remove audio input or not (ios just interrupts all your sessions if I remember correctly). Doing "pause" by ignoring both audio and video until interruption ends is a more generic solution.

Or to stop video recording as the native ios camera app does but this would require some additional event.

what additional event?

Event sent to dart side to inform it that video recording was stopped. So the dart side can handle that and somehow retrieve the resulting prematurely finished video file.

For this specific case would be enough to just call startRunning on capture sessions but that is too specific and would require ability to restart capture session from dart side as seems there is no good way how to test when such phone call or other such event ends, and trying to restart it for example every second inside FLTCam does not look like good idea.

startRunning on audio or video?

Probably startRunning on both video and audio as I mentioned if I remember correctly it interrupts all sessions.

Why requires ability to restart form dart side?

Because I did not find any sensible way to detect when it is "safe" to start sessions again. As I wrote it could try to do it every second until it succeeds but that looks like a bad solution. Instead maybe it is better to let users handle asynchronous errors from the dart side who can for example recreate the camera controller after clicking on a button or whatever looks like the best solution for a specific app.

Instead the camera controller enters an error state and can be recovered by disposing and recreating which should be more generic.

Is this what you plan to do or what the current state is?

Use single offset for both video and audio as separate offsets can cause unsynchronized video with audio.

Is this what you plan to do or what the current state is?

Current state both.

The rest is in updated description (I hope).