ryanheise
AudioSource.uri with headers not working when app is locked for some amount of time and audio is stopped
FIrst of all, thank you for this amazing package and all the other amazing packages.
Which API doesn't behave as documented, and how does it misbehave?
AudioSource.uri with headers present. When I try to add ANY headers to AudioSource.uri constructor and if I:
- Build playlist (add items to queue)
- Start playing audio
- Pause audio
- Lock screen
- Keep it locked for about 20 seconds
- Return to the app
- Skip queue items two or more times
Then this happens:
- Audio stops playing
- Player index quickly iterates through all remaining queue items and stops at the last one
- Audio player state is stuck to loading
Minimal reproduction project It's not exactly a fork, but here is the whole demo project with described exactly what is happeing including videos: https://github.com/invibe-sk/just_audio_flutter_app
To Reproduce (i.e. user steps, not code) Steps to reproduce the behavior: Please refer to videos and steps mentioned here: https://github.com/invibe-sk/just_audio_flutter_app
Error messages Not seeing any error messages, since the app is running in release or profile mode on real iOS device. I tried to hook up the Sentry, but no events get reported to the Sentry when this issue happens. This issue is not present on simulator.
Expected behavior After phone is locked for 20 seconds and then after the user returns to the app, I expect the rest of the playlist still remains playable with given headers applied.
Screenshots Please refer to videos and steps mentioned here: https://github.com/invibe-sk/just_audio_flutter_app
Smartphone (please complete the following information):
- Device: iPhone 12 Pro
- OS: iOS 15.4
Flutter SDK version
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.0.1, on macOS 12.3.1 21E258 darwin-arm, locale en-SK)
[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
[✓] Xcode - develop for iOS and macOS (Xcode 13.4)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2021.2)
[✓] VS Code (version 1.67.2)
[✓] Connected device (3 available)
[✓] HTTP Host Availability
• No issues found!
Additional context I also described this issue here: https://stackoverflow.com/questions/72425614/audiosource-uri-with-headers-not-working-when-app-is-locked-for-some-amount-of-t
I applied information from this guide when building the app: https://suragch.medium.com/background-audio-in-flutter-with-audio-service-and-just-audio-3cce17b4a7d
Thank you for any suggestions or replies.
EDIT: Also worth mentioning that when I initially build the playlist with ConcatenatingAudioSource, only the first two chapters (first two playlist / queue items) are requested on the server when the audio starts playing (according to nginx access log). The rest of the chapters (playlist / queue items) are requested as the player index keeps moving. Seems like some kind of lazy loading ? However, after the issue mentioned above (lock the screen, return to the app, trying to skip queue items then), nothing gets requested on the server. And only the initially requested chapters remains playable (the one on current index, and the next one - see the video2 in repo I provided).
We experience something similar on our end. It seems to be an issue that only exists on iOS, but I'm unable to reproduce this with a simulator and I don't own a physical iOS device at the moment.
From what I understand the steps to reproduce it are similar to the above:
- Play an item (in our case an HLS stream)
- Pause and put the app in the background
- Lock screen for about 30 seconds
- Open the app and try to play again
Apparently that causes an infinite loading loop and the playback cannot be started again (neither with the same file, nor with another audio file) until the app is restarted.
Trying to debug this without being able to reproduce it myself is difficult, but I've been able to at least get one error code from a user: HttpException: Bad file descriptor, uri = [our API server]/stream_0/s000040.ts
Hopefully I'll be able to borrow an iOS device soon to do some further testing, but any tips about what might be the reason for this is greatly appreciated!
I believe what is happening is that the proxy is being shut down (uncleanly) when the app is not in the foreground, and it is unable to resurrect the proxy. One workaround would be that once a player becomes unable to use its proxy for this reason, you can create a new player which will have a clean state.
There has been some work to improve the proxy's ability to detect network errors and restart itself, but there appear to still be some cases where it is unable to resurrect, and so a new player is the only way to reset the state.
I see, then creating a new player seems like what I have to do. How do I do that without running into Unhandled Exception: PlatformException(error, just_audio_background supports only a single player instance, null, null)?
~~I've tried awaiting dispose() before creating a new instance, but that does not seem to work.~~ Sorry, my bad. I accidentally ran it twice, that's why it didn't work.
If you are using just_audio_background, you don't have any control over such things, unless you edit the source code to make it work for your situation. If you were using audio_service directly, you would have more control over the creation of player instances in your app.
I see, I do need to use just_audio_background. So should I edit it to dispose and then re-initialize the background service on resume or what do you think is needed to get it to work with just_audio_background?
You don't need to re-initialise the service, you need to replace the player by a new player.
You're right, but I actually didn't have to customize the code to get it to work. Here's parts of the code that seems to be working for me:
@override
void didChangeAppLifecycleState(AppLifecycleState state) async {
if (state == AppLifecycleState.resumed) {
if (player.audioSource != null && !player.playing) {
await player.dispose();
player = AudioPlayer();
resumePlayer();
}
}
}
super.didChangeAppLifecycleState(state);
}
However, this resets it each time the app is resumed, regardless if there has been a proxy error or not. Any ideas about how to best tell if the player no longer is responding? I've tried to catch the exceptions and only run if there are exceptions, but it seems there aren't always exceptions. Might there be a timeout that hasn't been reached in those occasions?
Unfortunately not, if I knew how to detect this, it would be an easy next step to make an official fix. In the code, you'll see that there's a boolean flag inside the proxy that indicates whether the proxy is running and it is a matter of detecting this failure scenario and then setting this boolean to false. Of course, detecting it is the problem.
Of course, I should have guessed that. My current workaround is to detect when the app is paused (put in the background) and compare the difference in time when the app is resumed. If it's more than 30 seconds, then it reloads. The proxy doesn't fail if the screen is still on (from my testings) so it's not always needed, but this way I'm at least avoiding a reload if the user quickly switches between two apps and it still solves the issue in case of an actual proxy failure.
If you manage to resolve this properly at some point, do let me know.
I just realized that my workaround does not work when the user wants to resume the playback from the notification. Is there an easy way for me to remove the playback widget from the lock screen?
That way I can just force the user to open the app again if the app has been in the background for too long, i.e. when the proxy server has uncleanly been shut down.
I've had a look around in the source but so far I've had no success.
You can try calling stop or dispose to remove the notification. The actual code responsible for creating/destroying the notification is in the audio_service package.
That doesn't work unfortunately. This is what I'm attempting:
if (state == AppLifecycleState.paused && Platform.isIOS) {
resumed = false;
Future.delayed(const Duration(seconds: 30), () async {
if (!resumed && !player.playing) {
await player.stop();
// OR:
// await player.dispose();
}
});
}
The code is called properly but either way the notification is not closed.
The place to check if you were to dig into the code is audio_service/darwin/Classes/AudioServicePlugin.m if you search for the method name "stopService". Inserting a log in there may reveal whether or not that code is actually executing.
} else if ([@"stopService" isEqualToString:call.method]) {
[commandCenter.changePlaybackRateCommand setEnabled:NO];
[commandCenter.togglePlayPauseCommand setEnabled:NO];
[commandCenter.togglePlayPauseCommand removeTarget:nil];
[MPNowPlayingInfoCenter defaultCenter].nowPlayingInfo = nil;
processingState = ApsIdle;
actionBits = 0;
[self updateControls];
_controlsUpdated = NO;
startResult = nil;
commandCenter = nil;
result(@{});
}
After some initial testing, this seems to be a just_audio_background connected issue. If I run the example code in audio_service, it behaves as expected. However, running player.stop() within just_audio_background does not work.
Logs from the point in the code you referred to are printed when an item is loaded, but not when the stop method is actually called.
Thanks for reminding me. Can you test the fix/unimplemented_methods branch and see if it fixes your issue?
Afraid it didn't fix the issue for me, the notification is still there after calling stop() with that branch.
Not from what I can tell, it still seems to be called only when the item is loaded on my end.
I'm not quite sure if I understand your answer. You're answering "No, but sometimes yes". Is that correct? Can you describe the circumstance more clearly where stop is called?
Sorry, that was unclear. The stop code seems to be triggered when an item is loaded into the player, then I get the output in the logs that I inserted. After that, calling the actual stop method in just_audio does not print any new logs. So in other words, the code is never called when I want it to be called.
Stop is called on insert? That's strange, I am not aware of any code path from insert to stop. Unless an error has occurred during load, perhaps...
It is, I also noticed that before when I wanted to keep track of when the playback is started or stopped.
Running this code will print the log Start playback: false when an item is loaded, if you insert it into the example code in just_audio.
_player.playingStream.listen((event) {
print("Start playback: $event");
});
Here's the output from the log, not sure if that's enough for you. As you might see, I've also added logs for when the processing state changes. Other than that I get no error.
13:45:42.813497+0200 Runner <<<< FigFilePlayer >>>> playerfig_getNextPlaybackState: [0x14d652f50] P/BB called. reason ClientInitiated, options:
13:45:42.813569+0200 Runner <<<< FigFilePlayer >>>> playerfig_getNextPlaybackState: [0x14d652f50] P/BB player rate is 0. new playback state Paused
13:45:42.813592+0200 Runner <<<< FigFilePlayer >>>> playerfig_getNextPlaybackState: [0x14d652f50] P/BB new playback state: Paused (playerRate: 0.000), DON'T need to update item rate (nan). Previous state: Paused, change reason: ClientInitiated
13:45:42.813659+0200 Runner <<<< AVPlayerItem >>>> -[AVPlayerItem dealloc]: currentItem KVO: called
13:45:42.813699+0200 Runner <<<< AVPlayerItem >>>> -[AVPlayerItem dealloc]: currentItem KVO: I/JID.01
13:45:42.813684+0200 Runner <<<< FigPlayerSurrogate >>>> surrogatePlaybackItem_Invalidate: [0x6000015a9e00] I/JID.01 called
13:45:42.813772+0200 Runner <<<< FigFilePlayer >>>> itemfig_Invalidate: [0x1511ce800] I/JID.01 called
13:45:42.813797+0200 Runner <<<< FigFilePlayer >>>> playerfig_RemoveFromPlayQueue: [0x14d652f50] P/BB item 0x1511ce800 I/JID.01
13:45:42.813822+0200 Runner <<<< FigFilePlayer >>>> playerfig_maybeUndoQueueingForItem: [0x14d652f50] P/BB play queue before flush: [item 0x1511ce800 I/JID.01 (boss 0x14f84b730)]
13:45:42.813838+0200 Runner <<<< FigFilePlayer >>>> playerfig_maybeUndoQueueingForItem: [0x14d652f50] P/BB item 0x1511ce800 I/JID.01 is current item -- can't avoid a pause when current item has to be changed
13:45:42.813920+0200 Runner <<<< Boss >>>> FigPlaybackBossSetRateAndAnchorTime: (0x14f84b730) called, newRate = 0.000, itemTime = nan, hostClockTime = (now+) nan
13:45:42.813997+0200 Runner <<<< Boss >>>> bossScheduleReachedEndCallbackForTime: (0x14f84b730) called, endTime = nan
13:45:42.814090+0200 Runner <<<< Boss >>>> bossScheduleAdvanceForOverlappedPlaybackCallbackForTime: (0x14f84b730) called, advanceTime = nan
13:45:42.814193+0200 Runner <<< CFByteFlume >>> fbf_releaseInteractivePlaybackAssertion: [0x14d58a8a0] interactive playback count: 1 -> 0
13:45:42.814240+0200 Runner <<<< CRABS >>>> crabsReleaseReadAheadAssertion: [0x14fc2a0d0] read-ahead count: 1 -> 0
13:45:42.814316+0200 Runner <<<< Boss >>>> FigPlaybackBossSetRateAndAnchorTime: (0x14f84b730) called, newRate = 0.000, itemTime = nan, hostClockTime = (now+) nan
13:45:42.814342+0200 Runner <<<< Boss >>>> bossScheduleReachedEndCallbackForTime: (0x14f84b730) called, endTime = nan
13:45:42.814420+0200 Runner <<<< Boss >>>> bossScheduleAdvanceForOverlappedPlaybackCallbackForTime: (0x14f84b730) called, advanceTime = nan
13:45:42.814569+0200 Runner flutter: Start playback: false
13:45:42.814716+0200 Runner <<<< FigFilePlayer >>>> itemfig_vendAccessLogWhenItemStopsBeingCurrent: #Version: 1.0
#Software: AppleCoreMedia/1.0.0.19F70 (iPhone; U; CPU OS 15_5 like Mac OS X)
#Date: 2022/09/19 13:45:42.042
#Fields: date time uri cs-guid s-ip s-ip-changes sc-count c-duration-downloaded c-start-time c-duration-watched bytes c-observed-bitrate sc-indicated-bitrate c-stalls c-frames-dropped c-startup-time c-overdue c-reason c-observed-min-bitrate c-observed-max-bitrate c-observed-bitrate-sd s-playback-type sc-wwan-count c-switch-bitrate
2022/09/19 13:43:54.054 https://s3.amazonaws.com/scifri-episodes/scifri20181123-episode.mp3 F2E18676-...-...-...-... 52.217.135.56 1 4 - - - 32546811 7953863.854 - - - 1.337 - - - - - FILE 0 -
13:45:42.814730+0200 Runner flutter: ProcessingState.idle
13:45:42.814754+0200 Runner <<<< FigFilePlayer >>>> playerfig_gracefullyRemoveItemFromPlayQueue: [0x14d652f50] P/BB play queue now: []
13:45:42.814804+0200 Runner <<<< FigFilePlayer >>>> playerfig_stopResetDisturbReprepareAndResumeWithTransaction: [0x14d652f50] P/BB called for reason: playerfig_RemoveFromPlayQueue
13:45:42.814954+0200 Runner <<<< FAQ TIMING SHIM >>>> FigAudioQueueTimingShimStop: [0x6000012ac360] Calling AudioQueueStop(0x3e0f06e, immediate:1)
13:45:42.815182+0200 Runner <<<< FAQ TIMING SHIM >>>> FigAudioQueueTimingShimStop: [0x6000012ac360] Calling AudioQueueStop(0x3e0f06e, immediate:1)
13:45:42.815324+0200 Runner flutter: ProcessingState.idle
13:45:42.815330+0200 Runner <<<< FAQ >>>> subaq_disposeCAAudioQueue: [0x14d57dcb0] FigAudioQueueTimingShimDispose 0x6000012ac360
13:45:42.815351+0200 Runner <<<< FAQ TIMING SHIM >>>> FigAudioQueueTimingShimDispose: [0x6000012ac360] Calling AudioQueueDispose(0x3e0f06e, immediate:1)
13:45:42.815661+0200 Runner <<<< FigFilePlayer >>>> playerfig_getNextPlaybackState: [0x14d652f50] P/BB called. reason CurrentItemChanged, options:
13:45:42.815684+0200 Runner <<<< FigFilePlayer >>>> playerfig_getNextPlaybackState: [0x14d652f50] P/BB player rate is 0. new playback state Paused
13:45:42.815701+0200 Runner <<<< FigFilePlayer >>>> playerfig_getNextPlaybackState: [0x14d652f50] P/BB new playback state: Paused (playerRate: 0.000), DON'T need to update item rate (nan). Previous state: Paused, change reason: CurrentItemChanged
13:45:42.815783+0200 Runner <<<< FigPlayerSurrogate >>>> surrogatePlaybackItem_Invalidate: [0x6000015a9e00] I/JID.01 called
13:45:42.815845+0200 Runner <<< URLAsset >>> URLAssetFinalize: Called for asset 0x6000035d72c0
13:45:42.815895+0200 Runner flutter: ProcessingState.loading
13:45:42.815937+0200 Runner <<<< CRABS >>>> FigCRABSFinalize: [0x14fc2a0d0] Called
I've been trying to debug it to see if I can find anything else that could help us sort this out, but so far no luck. By the way, would you prefer that I open a new issue for this?
That could be helpful, particularly if you can provide a minimal reproduction project.
@ryanheise What I noticed on iOS is that including a header in the URL request causes an enormous spike in the CPU usage: From around 8% to 140%, with energy impact going from Low Impact to Very High. I think this might explain the issue mentioned in this ticket too.
Re. audio playback stopping on background; it seems on Android this is related to battery restrictions being enabled by default. Turning this to Unrestricted seems to 'resolve' the stopping of audio (though of course not good for battery life 😅 )
Regarding the behaviour of inserting through the entire playlist, you can test the treadmill branch which implements an experimental version of lazy loading for iOS.
I have implemented a feature in the feature/extractor_options branch that gives you the option to send headers natively on iOS rather than via the proxy. This uses the AVURLAssetHTTPHeaderFieldsKey key which is an undocumented API on iOS. It is for this reason that I have avoided it until now by going through a proxy. However, since Flutter's video_player plugin uses the AVURLAssetHTTPHeaderFieldsKey, I figure I should at least offer the option. To use it, set the useProxyForRequestHeaders constructor parameter to false.
Note, since this feature involves a change to the platform interface, in order to test it you will need to comment/uncomment lines in the pubspec as indicated. Here is a patch:
diff --git a/just_audio/example/pubspec.yaml b/just_audio/example/pubspec.yaml
index f187b4c..e39b7c9 100644
--- a/just_audio/example/pubspec.yaml
+++ b/just_audio/example/pubspec.yaml
@@ -18,11 +18,11 @@ dependencies:
path: ../
# Uncomment when testing platform interface changes.
-# dependency_overrides:
-# just_audio_platform_interface:
-# path: ../../just_audio_platform_interface
-# just_audio_web:
-# path: ../../just_audio_web
+dependency_overrides:
+ just_audio_platform_interface:
+ path: ../../just_audio_platform_interface
+ just_audio_web:
+ path: ../../just_audio_web
dev_dependencies:
# TODO: uncomment these dependencies when they have nullsafe versions.
diff --git a/just_audio/pubspec.yaml b/just_audio/pubspec.yaml
index 5d41e12..bff8530 100644
--- a/just_audio/pubspec.yaml
+++ b/just_audio/pubspec.yaml
@@ -14,12 +14,12 @@ environment:
flutter: ">=3.0.0"
dependencies:
- just_audio_platform_interface: ^4.2.2
- # just_audio_platform_interface:
- # path: ../just_audio_platform_interface
- just_audio_web: ^0.4.8
- # just_audio_web:
- # path: ../just_audio_web
+ # just_audio_platform_interface: ^4.2.2
+ just_audio_platform_interface:
+ path: ../just_audio_platform_interface
+ # just_audio_web: ^0.4.8
+ just_audio_web:
+ path: ../just_audio_web
audio_session: ^0.1.14
rxdart: '>=0.26.0 <0.28.0'
path: ^1.8.0
diff --git a/just_audio_background/example/pubspec.yaml b/just_audio_background/example/pubspec.yaml
index 1618cfb..d9a3463 100644
--- a/just_audio_background/example/pubspec.yaml
+++ b/just_audio_background/example/pubspec.yaml
@@ -16,11 +16,11 @@ dependencies:
path: ..
# Uncomment when testing platform interface changes.
-# dependency_overrides:
-# just_audio_platform_interface:
-# path: ../../just_audio_platform_interface
-# just_audio_web:
-# path: ../../just_audio_web
+dependency_overrides:
+ just_audio_platform_interface:
+ path: ../../just_audio_platform_interface
+ just_audio_web:
+ path: ../../just_audio_web
dev_dependencies:
# TODO: uncomment these dependencies when they have nullsafe versions.
diff --git a/just_audio_background/pubspec.yaml b/just_audio_background/pubspec.yaml
index 6f405a5..ed169d8 100644
--- a/just_audio_background/pubspec.yaml
+++ b/just_audio_background/pubspec.yaml
@@ -25,5 +25,5 @@ dev_dependencies:
flutter_lints: ^2.0.1
environment:
- sdk: ">=2.14.0 <4.0.0"
+ sdk: ">=2.17.0 <4.0.0"
flutter: ">=3.0.0"
diff --git a/just_audio_web/pubspec.yaml b/just_audio_web/pubspec.yaml
index bb096d9..02dfa08 100644
--- a/just_audio_web/pubspec.yaml
+++ b/just_audio_web/pubspec.yaml
@@ -11,9 +11,9 @@ flutter:
fileName: just_audio_web.dart
dependencies:
- just_audio_platform_interface: ^4.2.2
- # just_audio_platform_interface:
- # path: ../just_audio_platform_interface
+ # just_audio_platform_interface: ^4.2.2
+ just_audio_platform_interface:
+ path: ../just_audio_platform_interface
flutter:
sdk: flutter
flutter_web_plugins:
If it resolves the issue for you, please let me know and I can include it in the next release.
I've now published a new release which provides the option to pass useProxyForRequestHeaders: false into the constructor to get native header handling. This should sidestep any timeout issues with the proxy, although just be aware of the notes in the README about it using an undocumented API on the iOS side (on the other hand, video_player also uses the same API).