shaka-player icon indicating copy to clipboard operation
shaka-player copied to clipboard

Published 20 hours ago • verified publisher icon shaka-project

AdaptationSetCritieria extension to consider audio channel config rather than channel count

Open david-hm-morgan opened this issue 9 months ago • 0 comments

Have you read the FAQ and checked for duplicate open issues? Yes

Is your feature request related to a problem? Please describe.

I have been testing with some production content (which I cannot share the source of) but will describe and include some data below to set the scene.

There are several audio AdaptationSets representing two languages with stereo but two different audio codecs renditions, e.g.

  1. language 1, stereo, mp4a.40.2, audio channel config 2
  2. language 1, stereo, ac-3, audio channel config A000
  3. language 2, stereo, mp4a.40.2, audio channel config 2
  4. language 2, stereo, ac-3, audio channel config A000

Whilst language track selection is possible, and for stereo vs surround channels we can select and isolate tracks based on the channel count, it is not currently possible to isolate the stereo channel counts from each other. Shaka-Player treats all the ac-3 and mp4a.40.2 renditions as one adaptation set. I have concerns that audio gaps could happen during adaptation when you should only expect a small gap on track change.

The desire here would be to separate them based on their audio channel config and permit the user to select their choice of audio quality/codec.

Describe the solution you'd like

  1. PreferenceBasedCriteria addition:

    • static filterVariantsByAudioChannelConfig_(variants, audioChannelConfig) {}

  2. Potential addition of a config attribute alternative called preferredAudioChannelConfig default to 2.

  3. Extend the selectAudioLanguage() API to take an option audio channel config attribute as an alternative to channelsCount

    • selectAudioLanguage(language, role, channelsCount = 0, safeMargin = 0, audioChannelConfig = '') {}

Describe alternatives you've considered

Request content utilises the role attribute to differentiate alternative tracks.

Additional context

Here's the result of calling getVariantTracks() on an example stream.

Note I have change the language code to protect the obvious country of origin.

I have also preliminarily added support for audioChannelConfig into variant tracks, as a stepping stone to a potential solution and to demonstrate the different values that can hold. e.g.

  • 2
  • A000

As you can guess I have been already experimenting here and can continue to a full merge request, just wanting to gain buy in and also peer review for any gotchas I might have missed in this direction.

getVariantTracks() output 
[
  {
    "id": 0,
    "active": false,
    "type": "variant",
    "bandwidth": 6340000,
    "language": "cy",
    "label": "0",
    "kind": null,
    "width": 1920,
    "height": 1080,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.64002A, mp4a.40.2",
    "audioCodec": "mp4a.40.2",
    "videoCodec": "avc1.64002A",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 1,
    "audioId": 3,
    "channelsCount": 2,
    "audioChannelConfig": "2",
    "audioSamplingRate": 24000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 100000,
    "videoBandwidth": 6240000,
    "originalVideoId": "video_0",
    "originalAudioId": "audio513_5",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "cy"
  },
  {
    "id": 1,
    "active": true,
    "type": "variant",
    "bandwidth": 4280000,
    "language": "cy",
    "label": "0",
    "kind": null,
    "width": 1280,
    "height": 720,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.640029, mp4a.40.2",
    "audioCodec": "mp4a.40.2",
    "videoCodec": "avc1.640029",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 2,
    "audioId": 3,
    "channelsCount": 2,
    "audioChannelConfig": "2",
    "audioSamplingRate": 24000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 100000,
    "videoBandwidth": 4180000,
    "originalVideoId": "video_1",
    "originalAudioId": "audio513_5",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "cy"
  },
  {
    "id": 2,
    "active": false,
    "type": "variant",
    "bandwidth": 6340000,
    "language": "mul",
    "label": "0",
    "kind": null,
    "width": 1920,
    "height": 1080,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.64002A, mp4a.40.2",
    "audioCodec": "mp4a.40.2",
    "videoCodec": "avc1.64002A",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 1,
    "audioId": 4,
    "channelsCount": 2,
    "audioChannelConfig": "2",
    "audioSamplingRate": 24000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 100000,
    "videoBandwidth": 6240000,
    "originalVideoId": "video_0",
    "originalAudioId": "audio514_5",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "mul"
  },
  {
    "id": 3,
    "active": false,
    "type": "variant",
    "bandwidth": 4280000,
    "language": "mul",
    "label": "0",
    "kind": null,
    "width": 1280,
    "height": 720,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.640029, mp4a.40.2",
    "audioCodec": "mp4a.40.2",
    "videoCodec": "avc1.640029",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 2,
    "audioId": 4,
    "channelsCount": 2,
    "audioChannelConfig": "2",
    "audioSamplingRate": 24000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 100000,
    "videoBandwidth": 4180000,
    "originalVideoId": "video_1",
    "originalAudioId": "audio514_5",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "mul"
  },
  {
    "id": 4,
    "active": false,
    "type": "variant",
    "bandwidth": 6620000,
    "language": "cy",
    "label": "0",
    "kind": null,
    "width": 1920,
    "height": 1080,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.64002A, ac-3",
    "audioCodec": "ac-3",
    "videoCodec": "avc1.64002A",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 1,
    "audioId": 5,
    "channelsCount": 2,
    "audioChannelConfig": "A000",
    "audioSamplingRate": 48000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 380000,
    "videoBandwidth": 6240000,
    "originalVideoId": "video_0",
    "originalAudioId": "audio515_5",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "cy"
  },
  {
    "id": 5,
    "active": false,
    "type": "variant",
    "bandwidth": 4560000,
    "language": "cy",
    "label": "0",
    "kind": null,
    "width": 1280,
    "height": 720,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.640029, ac-3",
    "audioCodec": "ac-3",
    "videoCodec": "avc1.640029",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 2,
    "audioId": 5,
    "channelsCount": 2,
    "audioChannelConfig": "A000",
    "audioSamplingRate": 48000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 380000,
    "videoBandwidth": 4180000,
    "originalVideoId": "video_1",
    "originalAudioId": "audio515_5",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "cy"
  },
  {
    "id": 6,
    "active": false,
    "type": "variant",
    "bandwidth": 6640000,
    "language": "cy",
    "label": "0",
    "kind": null,
    "width": 1920,
    "height": 1080,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.64002A, ac-3",
    "audioCodec": "ac-3",
    "videoCodec": "avc1.64002A",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 1,
    "audioId": 6,
    "channelsCount": 2,
    "audioChannelConfig": "A000",
    "audioSamplingRate": 48000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 400000,
    "videoBandwidth": 6240000,
    "originalVideoId": "video_0",
    "originalAudioId": "audio515_2",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "cy"
  },
  {
    "id": 7,
    "active": false,
    "type": "variant",
    "bandwidth": 4580000,
    "language": "cy",
    "label": "0",
    "kind": null,
    "width": 1280,
    "height": 720,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.640029, ac-3",
    "audioCodec": "ac-3",
    "videoCodec": "avc1.640029",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 2,
    "audioId": 6,
    "channelsCount": 2,
    "audioChannelConfig": "A000",
    "audioSamplingRate": 48000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 400000,
    "videoBandwidth": 4180000,
    "originalVideoId": "video_1",
    "originalAudioId": "audio515_2",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "cy"
  },
  {
    "id": 8,
    "active": false,
    "type": "variant",
    "bandwidth": 6640000,
    "language": "mul",
    "label": "0",
    "kind": null,
    "width": 1920,
    "height": 1080,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.64002A, ac-3",
    "audioCodec": "ac-3",
    "videoCodec": "avc1.64002A",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 1,
    "audioId": 7,
    "channelsCount": 2,
    "audioChannelConfig": "A000",
    "audioSamplingRate": 48000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 400000,
    "videoBandwidth": 6240000,
    "originalVideoId": "video_0",
    "originalAudioId": "audio516_5",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "mul"
  },
  {
    "id": 9,
    "active": false,
    "type": "variant",
    "bandwidth": 4580000,
    "language": "mul",
    "label": "0",
    "kind": null,
    "width": 1280,
    "height": 720,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.640029, ac-3",
    "audioCodec": "ac-3",
    "videoCodec": "avc1.640029",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 2,
    "audioId": 7,
    "channelsCount": 2,
    "audioChannelConfig": "A000",
    "audioSamplingRate": 48000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 400000,
    "videoBandwidth": 4180000,
    "originalVideoId": "video_1",
    "originalAudioId": "audio516_5",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "mul"
  }
]

david-hm-morgan avatar May 23 '24 13:05 david-hm-morgan