Interfaces for "Enhanced UX Notification for Video and Audio Call Feature"

[satkh]

[Experimental] This Pull Request adds required interfaces for "Video Or Audio calling" feature on App Notifications.

Feature proposal link : https://github.com/microsoft/WindowsAppSDK/issues/4809

[satkh]

@satkh please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

[satkh]

@microsoft-github-policy-service agree [company="Microsoft"]

[satkh]

@microsoft-github-policy-service agree company="Microsoft"

[jonwis]

API design specs usually have clear "this is how an app would use it" samples. I see updates to tests, but not "Contoso wants to create a toast showing you this..."

[jonwis]

OK, so this feature is about "before you answer the call, configure your local devices in this way"? So I can see what I'm going to look like before I answer the call?

Is there another API coming that lets the application provide a video feed of the incoming caller so you can see who you're going to connect with?

[anupriya13]

API design specs usually have clear "this is how an app would use it" samples. I see updates to tests, but not "Contoso wants to create a toast showing you this..."

@jonwis Please review:

Existing Toast Schema: Toast schema - Windows UWP applications | Microsoft Learn

1 new XML tag to be introduced: <cameraPreview/>

• It will be under <visual> and will display only the camera preview
• If API is not called then system default to be displayed
• If camera load fails then it should not be displayed
• It should be linked to below new property in case user changes the device id it should display the preview of updated camera device id.
• Min Occurance = 0
• Max Occurance = 1
• Properties: None

1 new property to be introduced in existing <action/> tag: settingType="VideDevices" and settingType="AudioDevices"

• It is under <actions> and will include settings button
• If no devices available then it should not be visible
• Selected device ids will be the ids received from API if API is not invoked then system default values will be selected
• Existing Properties: https://learn.microsoft.com/en-us/uwp/schemas/tiles/toastschema/element-action
• Actions: https://learn.microsoft.com/en-us/uwp/schemas/tiles/toastschema/element-actions
• Additional new Properties:

  • settingType: optional, string

    When set to "videoDevices" or “audioDevices” the action becomes a setting action added to the toast notification with drop down options on devices list rather than a traditional toast button. Devices list includes either [camera, microphone and speakers] or [microphone and speakers]. settingType = “videoDevices” will show camera, mic and speaker list settingType= “audioDevices” will show mic and speaker list If placement="contextMenu" is also present, then this button won't be visible and will be ignored

Example of sending Video Call Notification using Toast XML (Contoso Calling App initiates a video call from user Jill Bender to user B sharing User B default App's device ids on the API):

void SendVideoCallNotification()  
{  
    winrt::hstring payload = LR"( 
<toast scenario="incomingCall" useButtonStyle="true">  
    <visual>  
        <binding template="ToastGeneric">  
            <text hint-maxLines="1">Jill Bender</text>  
            <text hint-maxLines="1">Incoming Video Call</text>  
           <cameraPreview/>  // NEW TAG!!
      </binding> 
    </visual> 
    <actions>  
        <action  
          content=""  
          imageUri="Assets/Icons/Setting.png"  
          settingType="videoDevices”   // NEW PROPERTY!!!
         arguments=""/>   
        <action  
          content=""  
          imageUri="Assets/Icons/Accept.png"  
         hint-buttonStyle="Success"  
          activationType="background"  
          arguments="action=reply&amp;threadId=92187"/>  
        <action  
          content=""  
          imageUri="Assets/Icons/Decline.png"  
          activationType="background"  
          hint-buttonStyle="Critical"  
          arguments="action=reply&amp;threadId=92187"/>  
         <action  
          content=""  
          imageUri="Assets/Icons/Message.png"  
          activationType="background"  
          arguments="action=reply&amp;threadId=92187"/>  
    </actions>  
</toast> 
)"; 

    winrt::AppNotification notification(payload);  
 
    if(winrt::AppNotificationDevicesData::IsVideoOrAudioCallingSupported()) 
    {     
       // Assign Devices Data values for the video call notification [New APIs Below]
       winrt::AppNotificationDevicesData devicesData; 

       devicesData.VideoDeviceId(L"\\?\USB#VID_045E&PID_0990&MI_00#6&db32c28&0&0000#{e5323777-f976-4f5b-9b55-b94699c46e44}\GLOBAL");  
 
       devicesData.AudioInputDeviceId(L"\\?\SWD#MMDEVAPI#{0.0.1.00000000}.{a19be0b4-e6e9-404b-b3ae-e98dc182e850}#{2eef81be-33fa-4800-9670-1cd474972c3f}");  
 
       devicesData.AudioOutputDeviceId(L"\\?\SWD#MMDEVAPI#{0.0.1.00000000}.{a19be0b4-e6e9-404b-b3ae-e98dc182e850}#{2eef81be-33fa-4800-9670-1cd474972c3f}"); 

       notification.DevicesData(devicesData); 
    } 
    winrt::AppNotificationManager::Default().Show(notification);  
} 

Example of sending Video Call Notification using AppNotificationBuilder (Contoso Calling App initiates a video call from user Jill Bender to user B sharing User B default App's device ids on the API):

void SendVideoCallNotification()  
{  
winrt::AppNotification notification =  
AppNotificationBuilder() 
    .SetScenario(AppNotificationScenario::IncomingCall) 
    .AddText(L"Jill Bender", AppNotificationTextProperties().SetMaxLines(1)) 
    .AddText(L"Incoming Video Call", AppNotificationTextProperties().SetMaxLines(1)) 
    .AddCameraPreview() // New API
    .AddButton(AppNotificationButton() 
        .SetIcon(winrt::Windows::Foundation::Uri(LR"(ms-appx://Assets/Icons/Setting.png)")) 
        .SetSettingType(AppNotificationSettingTypeButton::Devices)) // New API
    .AddButton(AppNotificationButton() 
        .AddArgument(L"action", L"acceptCall") 
        .AddArgument(L"threadId", L"92187") 
        .SetIcon(winrt::Windows::Foundation::Uri(LR"(ms-appx://Assets/Icons/Accept.png)")) 
        .SetButtonStyle(AppNotificationButtonStyle::Success)) 
    .AddButton(AppNotificationButton() 
        .AddArgument(L"action", L"declineCall") 
        .AddArgument(L"threadId", L"92187") 
        .SetIcon(winrt::Windows::Foundation::Uri(LR"(ms-appx://Assets/Icons/Decline.png)")) 
        .SetButtonStyle(AppNotificationButtonStyle::Critical)) 
    .AddButton(AppNotificationButton() 
        .AddArgument(L"action", L"message") 
        .AddArgument(L"threadId", L"92187") 
        .SetIcon(winrt::Windows::Foundation::Uri(LR"(ms-appx://Assets/Icons/Message.png)"))) 
    .BuildNotification(); 
    if(winrt::AppNotificationDevicesData::IsVideoOrAudioCallingSupported()) 
    {     
       // Assign Devices Data values for the video call notification  
       winrt::AppNotificationDevicesData devicesData; 

       devicesData.VideoDeviceId(L"\\?\USB#VID_045E&PID_0990&MI_00#6&db32c28&0&0000#{e5323777-f976-4f5b-9b55-b94699c46e44}\GLOBAL");  
 
       devicesData.AudioInputDeviceId(L"\\?\SWD#MMDEVAPI#{0.0.1.00000000}.{a19be0b4-e6e9-404b-b3ae-e98dc182e850}#{2eef81be-33fa-4800-9670-1cd474972c3f}");  
 
       devicesData.AudioOutputDeviceId(L"\\?\SWD#MMDEVAPI#{0.0.1.00000000}.{a19be0b4-e6e9-404b-b3ae-e98dc182e850}#{2eef81be-33fa-4800-9670-1cd474972c3f}");  

       notification.DevicesData(devicesData); 
    } 
    winrt::AppNotificationManager::Default().Show(notification);  
} 

Example of Invoke Callback Existing API with additional User Inputs key-values (Contoso Calling App will get the selected device ids from User B from below invoke API on any button clicked accept/ decline/ message):

//<toast >
//  <visual>
//      <binding template="ToastGeneric">
//          <text hint-maxLines="1">Jill Bender</text>
//      </binding>
//  </visual>
//  <actions>
//        <action  
//          content=""  
//          imageUri="Assets/Icons/Setting.png"  
//          settingType="VideoDevices”  // New Property
//          arguments=""/>   
 //       <action  
 //         content=""  
  //        imageUri="Assets/Icons/Accept.png"  
 //        hint-buttonStyle="Success"  
 //         activationType="background"  
 //         arguments="action=accept"/> 
//  </actions>
//</toast>

void ProcessNotificationArgs(const winrt::AppNotificationActivatedEventArgs& notificationActivatedEventArgs)
{
    // If the user clicks on a toast, the code will need to launch the chat thread window
    if (std::wstring(notificationActivatedEventArgs.Argument().c_str()).find(L"openThread") != std::wstring::npos)
    {
        GenerateChatThreadWindow();
    }
    else // If the user responds to a notification by clicking a accept button in the toast, we will need to initiate call to the other user
    if (std::wstring(notificationActivatedEventArgs.Argument().c_str()).find(L"accept") != std::wstring::npos)
    {
        auto input = notificationActivatedEventArgs.UserInput(); // Below 3 User Input key-values would be provided from OS 
        auto selectedCameraDeviceId  = input.Lookup(L"videoDeviceId");
        auto selectedMicrophoneDeviceId  = input.Lookup(L"audioInputDeviceId");
        auto selectedSpeakerDeviceId  = input.Lookup(L"audioOutputDeviceId");

        // Process the selected device ids and launch video / audio call 
        ActivateCallToUser(selectedCameraDeviceId  , selectedMicrophoneDeviceId  , selectedSpeakerDeviceId  );
    }
}

OK, so this feature is about "before you answer the call, configure your local devices in this way"? So I can see what I'm going to look like before I answer the call?

Yes

Is there another API coming that lets the application provide a video feed of the incoming caller so you can see who you're going to connect with?

Not yet, we don't have it in current scope.

[anupriya13]

Let's wait for FrameworkUDK changes to be checked in here before merging

[satkh]

Please fix the tooltip topic before you merge. The "be inert on unsupported" is optional to fix.

This "m_toolTip" is not added as part of this feature its existing. I see those special characters (<,>,',") already escaped in the "set" method below.

[satkh]

/azp run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 1 pipeline(s).

[satkh]

/azp run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 1 pipeline(s).