Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AdaptationSetCritieria extension to consider audio channel config rather than channel count #6658

Closed
david-hm-morgan opened this issue May 23, 2024 · 2 comments
Closed

AdaptationSetCritieria extension to consider audio channel config rather than channel count #6658

david-hm-morgan opened this issue May 23, 2024 · 2 comments
Labels
status: archived Archived and locked; will not be updated type: enhancement New feature or request

Comments

@david-hm-morgan
Copy link
Contributor

Have you read the FAQ and checked for duplicate open issues?
Yes

Is your feature request related to a problem? Please describe.

I have been testing with some production content (which I cannot share the source of) but will describe and include some data below to set the scene.

There are several audio AdaptationSets representing two languages with stereo but two different audio codecs renditions, e.g.

  1. language 1, stereo, mp4a.40.2, audio channel config 2
  2. language 1, stereo, ac-3, audio channel config A000
  3. language 2, stereo, mp4a.40.2, audio channel config 2
  4. language 2, stereo, ac-3, audio channel config A000

Whilst language track selection is possible, and for stereo vs surround channels we can select and isolate tracks based on the channel count, it is not currently possible to isolate the stereo channel counts from each other. Shaka-Player treats all the ac-3 and mp4a.40.2 renditions as one adaptation set. I have concerns that audio gaps could happen during adaptation when you should only expect a small gap on track change.

The desire here would be to separate them based on their audio channel config and permit the user to select their choice of audio quality/codec.

Describe the solution you'd like

  1. PreferenceBasedCriteria addition:

    • static filterVariantsByAudioChannelConfig_(variants, audioChannelConfig) {}

  2. Potential addition of a config attribute alternative called preferredAudioChannelConfig default to 2.

  3. Extend the selectAudioLanguage() API to take an option audio channel config attribute as an alternative to channelsCount

    • selectAudioLanguage(language, role, channelsCount = 0, safeMargin = 0, audioChannelConfig = '') {}

Describe alternatives you've considered

Request content utilises the role attribute to differentiate alternative tracks.

Additional context

Here's the result of calling getVariantTracks() on an example stream.

Note I have change the language code to protect the obvious country of origin.

I have also preliminarily added support for audioChannelConfig into variant tracks, as a stepping stone to a potential solution and to demonstrate the different values that can hold. e.g.

  • 2
  • A000

As you can guess I have been already experimenting here and can continue to a full merge request, just wanting to gain buy in and also peer review for any gotchas I might have missed in this direction.

getVariantTracks() output 
[
  {
    "id": 0,
    "active": false,
    "type": "variant",
    "bandwidth": 6340000,
    "language": "cy",
    "label": "0",
    "kind": null,
    "width": 1920,
    "height": 1080,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.64002A, mp4a.40.2",
    "audioCodec": "mp4a.40.2",
    "videoCodec": "avc1.64002A",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 1,
    "audioId": 3,
    "channelsCount": 2,
    "audioChannelConfig": "2",
    "audioSamplingRate": 24000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 100000,
    "videoBandwidth": 6240000,
    "originalVideoId": "video_0",
    "originalAudioId": "audio513_5",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "cy"
  },
  {
    "id": 1,
    "active": true,
    "type": "variant",
    "bandwidth": 4280000,
    "language": "cy",
    "label": "0",
    "kind": null,
    "width": 1280,
    "height": 720,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.640029, mp4a.40.2",
    "audioCodec": "mp4a.40.2",
    "videoCodec": "avc1.640029",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 2,
    "audioId": 3,
    "channelsCount": 2,
    "audioChannelConfig": "2",
    "audioSamplingRate": 24000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 100000,
    "videoBandwidth": 4180000,
    "originalVideoId": "video_1",
    "originalAudioId": "audio513_5",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "cy"
  },
  {
    "id": 2,
    "active": false,
    "type": "variant",
    "bandwidth": 6340000,
    "language": "mul",
    "label": "0",
    "kind": null,
    "width": 1920,
    "height": 1080,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.64002A, mp4a.40.2",
    "audioCodec": "mp4a.40.2",
    "videoCodec": "avc1.64002A",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 1,
    "audioId": 4,
    "channelsCount": 2,
    "audioChannelConfig": "2",
    "audioSamplingRate": 24000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 100000,
    "videoBandwidth": 6240000,
    "originalVideoId": "video_0",
    "originalAudioId": "audio514_5",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "mul"
  },
  {
    "id": 3,
    "active": false,
    "type": "variant",
    "bandwidth": 4280000,
    "language": "mul",
    "label": "0",
    "kind": null,
    "width": 1280,
    "height": 720,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.640029, mp4a.40.2",
    "audioCodec": "mp4a.40.2",
    "videoCodec": "avc1.640029",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 2,
    "audioId": 4,
    "channelsCount": 2,
    "audioChannelConfig": "2",
    "audioSamplingRate": 24000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 100000,
    "videoBandwidth": 4180000,
    "originalVideoId": "video_1",
    "originalAudioId": "audio514_5",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "mul"
  },
  {
    "id": 4,
    "active": false,
    "type": "variant",
    "bandwidth": 6620000,
    "language": "cy",
    "label": "0",
    "kind": null,
    "width": 1920,
    "height": 1080,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.64002A, ac-3",
    "audioCodec": "ac-3",
    "videoCodec": "avc1.64002A",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 1,
    "audioId": 5,
    "channelsCount": 2,
    "audioChannelConfig": "A000",
    "audioSamplingRate": 48000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 380000,
    "videoBandwidth": 6240000,
    "originalVideoId": "video_0",
    "originalAudioId": "audio515_5",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "cy"
  },
  {
    "id": 5,
    "active": false,
    "type": "variant",
    "bandwidth": 4560000,
    "language": "cy",
    "label": "0",
    "kind": null,
    "width": 1280,
    "height": 720,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.640029, ac-3",
    "audioCodec": "ac-3",
    "videoCodec": "avc1.640029",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 2,
    "audioId": 5,
    "channelsCount": 2,
    "audioChannelConfig": "A000",
    "audioSamplingRate": 48000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 380000,
    "videoBandwidth": 4180000,
    "originalVideoId": "video_1",
    "originalAudioId": "audio515_5",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "cy"
  },
  {
    "id": 6,
    "active": false,
    "type": "variant",
    "bandwidth": 6640000,
    "language": "cy",
    "label": "0",
    "kind": null,
    "width": 1920,
    "height": 1080,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.64002A, ac-3",
    "audioCodec": "ac-3",
    "videoCodec": "avc1.64002A",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 1,
    "audioId": 6,
    "channelsCount": 2,
    "audioChannelConfig": "A000",
    "audioSamplingRate": 48000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 400000,
    "videoBandwidth": 6240000,
    "originalVideoId": "video_0",
    "originalAudioId": "audio515_2",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "cy"
  },
  {
    "id": 7,
    "active": false,
    "type": "variant",
    "bandwidth": 4580000,
    "language": "cy",
    "label": "0",
    "kind": null,
    "width": 1280,
    "height": 720,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.640029, ac-3",
    "audioCodec": "ac-3",
    "videoCodec": "avc1.640029",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 2,
    "audioId": 6,
    "channelsCount": 2,
    "audioChannelConfig": "A000",
    "audioSamplingRate": 48000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 400000,
    "videoBandwidth": 4180000,
    "originalVideoId": "video_1",
    "originalAudioId": "audio515_2",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "cy"
  },
  {
    "id": 8,
    "active": false,
    "type": "variant",
    "bandwidth": 6640000,
    "language": "mul",
    "label": "0",
    "kind": null,
    "width": 1920,
    "height": 1080,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.64002A, ac-3",
    "audioCodec": "ac-3",
    "videoCodec": "avc1.64002A",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 1,
    "audioId": 7,
    "channelsCount": 2,
    "audioChannelConfig": "A000",
    "audioSamplingRate": 48000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 400000,
    "videoBandwidth": 6240000,
    "originalVideoId": "video_0",
    "originalAudioId": "audio516_5",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "mul"
  },
  {
    "id": 9,
    "active": false,
    "type": "variant",
    "bandwidth": 4580000,
    "language": "mul",
    "label": "0",
    "kind": null,
    "width": 1280,
    "height": 720,
    "frameRate": null,
    "pixelAspectRatio": null,
    "hdr": null,
    "videoLayout": null,
    "mimeType": "video/mp4",
    "audioMimeType": "audio/mp4",
    "videoMimeType": "video/mp4",
    "codecs": "avc1.640029, ac-3",
    "audioCodec": "ac-3",
    "videoCodec": "avc1.640029",
    "primary": false,
    "roles": [],
    "audioRoles": [],
    "forced": false,
    "videoId": 2,
    "audioId": 7,
    "channelsCount": 2,
    "audioChannelConfig": "A000",
    "audioSamplingRate": 48000,
    "spatialAudio": false,
    "tilesLayout": null,
    "audioBandwidth": 400000,
    "videoBandwidth": 4180000,
    "originalVideoId": "video_1",
    "originalAudioId": "audio516_5",
    "originalTextId": null,
    "originalImageId": null,
    "originalLanguage": "mul"
  }
]
@david-hm-morgan david-hm-morgan added the type: enhancement New feature or request label May 23, 2024
@shaka-bot shaka-bot added this to the Backlog milestone May 23, 2024
@avelad
Copy link
Member

avelad commented May 27, 2024

Are you planning send a PR to add it? Thanks!
@avelad avelad added the status: waiting on response Waiting on a response from the reporter(s) of the issue label May 27, 2024
@shaka-bot
Copy link
Collaborator

Closing due to inactivity. If this is still an issue for you or if you have further questions, the OP can ask shaka-bot to reopen it by including @shaka-bot reopen in a comment.
@shaka-bot shaka-bot removed the status: waiting on response Waiting on a response from the reporter(s) of the issue label Jun 3, 2024
@avelad avelad removed this from the Backlog milestone Jun 3, 2024
@shaka-bot shaka-bot added the status: archived Archived and locked; will not be updated label Aug 2, 2024
@shaka-project shaka-project locked as resolved and limited conversation to collaborators Aug 2, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
status: archived Archived and locked; will not be updated type: enhancement New feature or request
3 participants
@avelad @shaka-bot @david-hm-morgan