Media responses: can the user-interrupted playback be resumed?

[solyarisoftware]

trafficstars

Or: Media responses: Handling callback BEFORE playback completion?

The issue is related to the closed issue: https://github.com/actions-on-google/actions-on-google-nodejs/issues/243

The UX scenario: I experienced an issue, when trying to develop an action on google application (using Action SDK) to replicate google assistant UX when a user listen a Spotify playlist...

Specifically I would let the user interrupt the audio streaming asking something about what listening in a specific moment. That's appears not possible so far.

I would like to play some audio file (media response), say a MP3 song long something like three minutes (00:03:00). Afterward, at the middle of the playing song, say at 00:01:45, I would let user ask for example:

Ok Google, What's the name of the song?

or

Ok Google, Who play the Rhodes piano?

The actions would manage a Q/A, with these possible steps:

• PAUSE the playback (implicit)
• catch the user intent,
• reply user with the answer
• RESUME the playback (TBD).

My test result fails: When the media response plays, the action is able to interrupt the audio and it's able manage the question, but the playback IS NOT RESUMED. That's weird (for my ears)!

It's debatable, but I think that's a BUG, or at least let's call a Change Request: I would let developer the choice to resume or not the underlying streaming after a user interruption.

BTW, The fact interrupted audio streaming is resumed after a user Q/A, is in compliance with what happens on Google Assistant/Google Home with live streaming or an long running playlist streaming. By example, if a Google Home user say

Ehi Google, play BBC Radio 4 blablabla brexit ... blablabla brexit ... blablabla brexit ...

// interrupting the BBC streaming with a question: Ehi Google, what's the weather? Right now in Genoa, it's sunny blablabla ...

// end the answer, RESUME the BBC streaming blablabla brexit ... blablabla brexit ... blablabla brexit ...

// until the user explicitly ask to close Ehi Google, stop

---

So, INTO THE THIRD PARTY APPS (ACTIONS) I would like the same UX for the final user. The listener of an audio file (song/news/book, whatever) would be able to interrupt the audio streaming, ask a question and afterward the audio streaming would (optionally) be RESUMED.

So my request, from the developer perspective, is related on that possibility to OPT-IN an audio playback resume mode. My proposal is:

• to modify behavior of actions.intent.MEDIA_STATUS to catch the interrupt status (let's call it STOPPED).
• allowing to playback an audio file from an offset time (to enable 'resume')

The event BEFORE playback completion would report the elapsed time from the start time of audio streaming (e.g. 00:01:45, or in milliseconds amount), to allow developer to understand the exact point of user interrupt. That's important to answer user on the precise context (e.g. in case audio was an audio book, or a news, etc.).

See the following PSEUDO code:

/*
 * MEDIA_STATUS intent
 * Track the elapsed time (milliseconds) until user stop playback
 */
app.intent('actions.intent.MEDIA_STATUS', (conv) => {

  const mediaStatus = conv.arguments.get('MEDIA_STATUS')
  
  // CHANGE REQUEST
  // suppose the event: 'STOPPED'
  if (mediaStatus && mediaStatus.status === 'STOPPED') {
     
     // memorize number of milliseconds playd until the user stop.
      offsetTime = mediaStatus.stopTime 
   }      
  else if (mediaStatus && mediaStatus.status === 'FINISHED') {
    response = 'Hope you enjoyed the tune!'
  }
}

/*
  * ON-PLAYBACK QUERIES intent
  */
app.intent('actions.intent.TEXT', (conv) => {

     // reply to query 
     // parse the query text and find an answer on a DB, etc.
     ...

    // CHANGE REQUEST
    // resume the playback after the reply
    // resuming playback from the offsetTime  
    // suppose having the method resume()
     conv.resume(mediaresponse, offsetTime)
}

A last note: The request for me is crucial to develop a new generation of conversational UX, where user can interact with a "static" content streaming (music, book, news, any audio file or streaming) interacting and conversating with that "content"... (using metadata, etc,etc. ;-) ).

Please let me know if there is something obscure in my rant or I missing something about an already possible configuration.

If you like my request, please vote +1

Thanks for your patience. giorgio

[solyarisoftware]

I edited the original text to better explain my proposal, adding pseudocode.

[Fleker]

I can reproduce this behavior with resuming not working. Additionally, as one cannot get the playback offset or set it, it would be difficult to resume it yourself. I will file a bug internally.

[solyarisoftware]

Thanks Nick!

The minimal request could be:

1. set as default behavior: when the playback is interrupted (== stopped) by any user request, the playback is resumed (from interrupt offset) after the action reply to user, with app.ask(reply); or better: resuming playback when developer state explicitly:

app.resume(mediaResponse)

BTW In this solution the offset remains "hidden" :(

A maybe better solution:

2. Let developer get/set the offset.

• When the stopped event is detected' actions.intent.MEDIA_STATUS could return the offset.
• Afterward developer will optionally resume with

app.resume(mediaResponse, offset)

BTW, developer could optionally resume the track from a different offset value.

That's important to "interact" on audio content if it would be associated with some timestamping & metadata info;

00:00:00 Guitar intro
00:00:23 Guitar, Rhodes, Drums & Bass
00:01:02 Rhodes solo
00:02:00 blablabla