Lock Screen Controls and Metadata Support for Windows, Android, IOS and Mac Catalyst

[ne0rrmatrix]

• Feature/Proposal

Add Support for Lock Screen Controls and Metadata.

Description of Change

Add Lock Screen Media Controls to Windows, Android, iOS, and Mac Catalyst. Metadata like Title, Artist, and Album art will be displayed for playing media.

Linked Issues

• Fixes #1682
• Fixes #1667
• Fixes #1765

PR Checklist

• [x] Has a linked Issue, and the Issue has been approved(bug) or Championed (feature/proposal)
• [ ] Has tests (if omitted, state reason in description)
• [x] Has samples (if omitted, state reason in description)
• [x] Rebased on top of main at time of PR
• [x] Changes adhere to coding standard
• [x] Documentation created or updated: https://github.com/MicrosoftDocs/CommunityToolkit/pull/414

Additional information

Add support for Lock Screen Controls to Windows, Mac, IOS, and Android. Support for adding Metadata in XAML or code behind.

Example usage in XAML:

 <toolkit:MediaElement
     x:Name="MediaElement"
     WidthRequest="{OnIdiom Tablet=410, Default=380}"
     ShouldAutoPlay="True"
     Source="https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4"
     MetaDataArtworkUrl="https://michellevella.com/cdn/shop/products/BeatlesLonelyHearts4UP_1728x.jpg?v=1660582971"
     MetaDataTitle="Big Buck Bunny"
     MetaDataArtist="Blender Foundation"
     SourceType="Audio"
   MediaEnded="OnMediaEnded"
     MediaFailed="OnMediaFailed"
     MediaOpened="OnMediaOpened"
     PositionChanged="OnPositionChanged"
     StateChanged="OnStateChanged"
     SeekCompleted="OnSeekCompleted"/>

Example in code behind:

async void ChangeSourceClicked(Object sender, EventArgs e)
{
	var result = await DisplayActionSheet("Choose a source", "Cancel", null,
		loadOnlineMp4, loadHls, loadLocalResource, resetSource);

	switch (result)
	{
		case loadOnlineMp4:
			MediaElement.MetaDataTitle = "Big Buck Bunny";
			MediaElement.MetaDataArtworkUrl = "https://michellevella.com/cdn/shop/products/BeatlesLonelyHearts4UP_1728x.jpg?v=1660582971";
			MediaElement.MetaDataArtist = "Big Buck Bunny Album";
			MediaElement.SourceType = Primitives.MediaElementSourceType.Audio;
			MediaElement.Source =
				MediaSource.FromUri(
					"https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4");
			return;

		case loadHls:
			MediaElement.MetaDataArtist = "HLS Album";
			MediaElement.MetaDataArtworkUrl = "https://michellevella.com/cdn/shop/products/BeatlesLonelyHearts4UP_1728x.jpg?v=1660582971";
			MediaElement.MetaDataTitle = "HLS Title";
			MediaElement.SourceType = Primitives.MediaElementSourceType.Audio;
			MediaElement.Source
				= MediaSource.FromUri(
					"https://mtoczko.github.io/hls-test-streams/test-gap/playlist.m3u8");
			return;

		case resetSource:
			MediaElement.MetaDataArtworkUrl = string.Empty;
			MediaElement.MetaDataTitle = string.Empty;
			MediaElement.MetaDataArtist = string.Empty;
			MediaElement.SourceType = Primitives.MediaElementSourceType.Unknown;
			MediaElement.Source = null;
			return;

		case loadLocalResource:
			MediaElement.MetaDataArtworkUrl = "https://michellevella.com/cdn/shop/products/BeatlesLonelyHearts4UP_1728x.jpg?v=1660582971";
			MediaElement.MetaDataTitle = "Local Resource Title";
			MediaElement.MetaDataArtist = "Local Resource Album";
			MediaElement.SourceType = Primitives.MediaElementSourceType.Audio;

			if (DeviceInfo.Platform == DevicePlatform.MacCatalyst
				|| DeviceInfo.Platform == DevicePlatform.iOS)
			{
				MediaElement.Source = MediaSource.FromResource("AppleVideo.mp4");
			}
			else if (DeviceInfo.Platform == DevicePlatform.Android)
			{
				MediaElement.Source = MediaSource.FromResource("AndroidVideo.mp4");
			}
			else if (DeviceInfo.Platform == DevicePlatform.WinUI)
			{
				MediaElement.Source = MediaSource.FromResource("WindowsVideo.mp4");
			}
			return;
	}
}

Sample API

/// <summary>
/// Gets or sets the Title of the media.
/// This is a bindable property.
/// </summary>
public string MetaDataTitle
{
	get => (string)GetValue(MetaDataTitleProperty);
	set => SetValue(MetaDataTitleProperty, value);
}

/// <summary>
/// Gets or sets the Artist of the media.
/// This is a bindable property.
/// </summary>
public string MetaDataArtist
{
	get => (string)GetValue(MetaDataArtistProperty);
	set => SetValue(MetaDataArtistProperty, value);
}

/// <summary>
/// Gets or sets the Artwork Image Url of the media.
/// This is a bindable property.
/// </summary>
public string MetaDataArtworkUrl
{
	get => (string)GetValue(MetaDataArtworkUrlProperty);
	set => SetValue(MetaDataArtworkUrlProperty, value);
}

/// <summary>
/// Gets or sets the source type of the media.
/// This is a bindable property.
/// </summary>
public MediaElementSourceType SourceType
{
	get => (MediaElementSourceType)GetValue(MediaElementSourceTypeProperty);
	set => SetValue(MediaElementSourceTypeProperty, value);
}

Android and Windows

iOS

Mac Catalyst

[brminnick]

Correct. Every Task should be awaited so that Exceptions that occur inside the Task are re-thrown by the .NET runtime.

Discarding a Task, eg _ = SomeAsyncMethod(), is not a best practice.

I have a video that dives deeper into this topic here: https://youtu.be/GQYd6MWKiLI?si=B8OdPaIyIeUyu5Q2

[ne0rrmatrix]

Correct. Every Task should be awaited so that Exceptions that occur inside the Task are re-thrown by the .NET runtime.

Discarding a Task, eg _ = SomeAsyncMethod(), is not a best practice.

I have a video that dives deeper into this topic here: https://youtu.be/GQYd6MWKiLI?si=B8OdPaIyIeUyu5Q2

I will spend some time today and tomorrow going through the code and fixing this. Thank you for the video link. I will make sure to watch it and make the appropriate changes in the code afterwards.

[ne0rrmatrix]

Thanks for this huge feature! I saw a lot of Task methods being ignored (_ = SomeAsyncMethod();), don't we want to provide some feedback if they fail?

Can you look at the new changes and see if I did async - await correctly. I tried to follow your suggestions in the video you linked. @brminnick @pictos

[ne0rrmatrix]

There're some changes needed on the async part... Let me know if I can help you better

If the changes I made in response to your latest requests are not good I would appreciate any suggestions as to how to fix it.

[ne0rrmatrix]

@ne0rrmatrix, I took the liberty of taking this and moving forward, so I refactor all the code related to the async tasks, that way I hope to unblock you and this PR. Let me know if you've any question on what I did, I'll be more than happy to provide further context.

Not related to this PR, but while working on it I just saw that ExoPlayer is obsolete, they moved it to AndroidX lib, more info here, should we look and migrate to this lib?

@pictos thank you for fixing all the code. I appreciate it. I will look and try to remember the pattern for async await that you have implemented. You did great! BTW we are currently blocked for updating to AndroidX as the bindings are not out yet. But there is an open PR with regards to this that should start the process of migration once the bindings are available. Or at least I saw a team member had started a discussion about it with regards to media element.

[ne0rrmatrix]

Before we pull the trigger does anyone want to update the doc's with how to add metadata? I can do that or does someone who is good with Markdown want to do it?

[ne0rrmatrix]

At a minimum the docs need to be updated for android manifest changes. plist changes for mac and iOS needs to be done too. Updating the docs for how to use and set the metadata would be useful too.

[ne0rrmatrix]

Updated docs. Link to PR: https://github.com/MicrosoftDocs/CommunityToolkit/pull/414

[brminnick]

Awesome!! I've added the https://github.com/CommunityToolkit/Maui/labels/approved label and temporarily added the https://github.com/CommunityToolkit/Maui/labels/pending%20documentation label. Once Shaun has approved the docs, we'll merge this PR 🚀

I've also added the https://github.com/CommunityToolkit/Maui/labels/needs%20discussion label so that we can highlight it in this week's standup!

[ne0rrmatrix]

IOS and Mac changes tested on Mac Mini M2 and on 10 generation Ipad as working with NStransport removed.

[YehonatanVishna]

When would I be able to use the new player with metadata?

[jfversluis]

@YehonatanVishna right after merging this we found a bug we wanted to fix first. I think that has happened by now and we should be able to release. I will double check and probably, hopefully, release something in the upcoming week.

thanks!

[brminnick]

@YehonatanVishna right after merging this we found a bug we wanted to fix first. I think that has happened by now and we should be able to release. I will double check and probably, hopefully, release something in the upcoming week.

Yup! Both CommunityToolkit.Maui.Camera v1.0.0 and CommunityToolkit.Maui.MediaElement v4.0.1 @jfversluis!

We should probably copy/paste the release notes from CommunityToolkit.Maui.MediaElement v4.0.0 into v4.0.1 just so folks don't miss the new Lock Screen feature.

[YehonatanVishna]

Additionally, I tried testing this new feature by using the merged code. I noticed 2 things:

1. The metadata doesn't work at all for me in android (although I need to mention it works great on windows)
2. The MediaElement.SourceType property doesn't seem to exist (at least I could not use it).

[ne0rrmatrix]

Additionally, I tried testing this new feature by using the merged code. I noticed 2 things:

1. The metadata doesn't work at all for me in android (although I need to mention it works great on windows)
2. The MediaElement.SourceType property doesn't seem to exist (at least I could not use it).

For metadata image it needs to be a url. For the other two fields it should be text. The docs show example as does the sample app. When I get home I will look into the android issue. Or at least it will once merged. You can look at sample app in main for now.

[brminnick]

@ne0rrmatrix would it make sense to submit a PR that changes the type of the MetadataImage to URI before we realest v4.0.1?

Are there any scenarios where we wouldn't use a URL?

[ne0rrmatrix]

@ne0rrmatrix would it make sense to submit a PR that changes the type of the MetadataImage to URI before we realest v4.0.1?

Are there any scenarios where we wouldn't use a URL?

The artwork uses a url. The other two are strings. It currently only supports those formats.

[YehonatanVishna]

What about the issue with SourceType?

[ne0rrmatrix]

What about the issue with SourceType?

I will look into it when I get home this weekend.

[bijington]

What about the issue with SourceType?

There is no SourceType property on MediaElement, where are you trying to use it? Is there anything to suggest it should exist or is this something extra you are asking for?

[YehonatanVishna]

Nevermind, I read it in your original code example (from this pr), but it seems not be in - the code. I assume you removed it.

[ne0rrmatrix]

Additionally, I tried testing this new feature by using the merged code. I noticed 2 things:

1. The metadata doesn't work at all for me in android (although I need to mention it works great on windows)
2. The MediaElement.SourceType property doesn't seem to exist (at least I could not use it).

For android you need to modify the manifest file

1. Add ResizableActivity and Launchmode to Activity

[Activity(Theme = "@style/Maui.SplashTheme", ResizeableActivity = true,  MainLauncher = true, LaunchMode = LaunchMode.SingleTask)]
public class MainActivity : MauiAppCompatActivity
{
}

2. Add the following to AndroidManifest.xml inside the <application> tag.

 <service android:name="CommunityToolkit.Maui.Media.Services" android:exported="false" android:enabled="true" android:foregroundServiceType="mediaPlayback">
   <intent-filter>
     <action android:name="android.intent.action.MEDIA_BUTTON" />
   </intent-filter>
   <intent-filter>
     <action android:name="androidx.media3.session.MediaSessionService"/>
   </intent-filter>
 </service>

3. Add the following permissions to AndroidManifest.xml

<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
<uses-permission android:name="android.permission.POST_NOTIFICATIONS" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MEDIA_PLAYBACK" />
<uses-permission android:name="android.permission.MEDIA_CONTENT_CONTROL" />

Here is an example of required settings in AndroidManifest.xml

<application android:allowBackup="true" android:icon="@mipmap/appicon" android:enableOnBackInvokedCallback="true" android:supportsRtl="true">
<service android:name="CommunityToolkit.Maui.Media.Services" android:exported="false" android:enabled="true" android:foregroundServiceType="mediaPlayback">
    <intent-filter>
    <action android:name="android.intent.action.MEDIA_BUTTON" />
    </intent-filter>
    <intent-filter>
    <action android:name="androidx.media3.session.MediaSessionService"/>
    </intent-filter>
</service>
</application>
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
<uses-permission android:name="android.permission.POST_NOTIFICATIONS" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MEDIA_PLAYBACK" />
<uses-permission android:name="android.permission.MEDIA_CONTENT_CONTROL" />

[!NOTE] This modification to the Android manifest enables metadata display when playing a video. It provides support for notifications and is essential for notifications to function across all relevant APIs. The change introduces a service and grants necessary permissions.

[brminnick]

Thanks @ne0rrmatrix! Just checking - is all of this documented in our docs? (CC @bijington) https://github.com/MicrosoftDocs/CommunityToolkit/pull/414

[ne0rrmatrix]

Thanks @ne0rrmatrix! Just checking - is all of this documented in our docs? (CC @bijington) MicrosoftDocs/CommunityToolkit#414

@brminnick yes it is all in docs. Waiting to be merged when it is released. What i quoted above was directly from updated docs.

[kenboy2304]

"I have tried using MetaData and granted permissions for both Android and iOS. Currently, the Control only appears on the lock screen of Android, but it does not appear on iOS."

[jfversluis]

@kenboy2304 please open a new issue with more details, closed issues and merged PRs are not monitored. If you want us to help, we will need a lot more detail than you're giving here :)