NewPipe
NewPipe copied to clipboard
TeamNewPipe
YouTube: Support multiple audio tracks
Checklist
- [x] I checked, but didn't find any duplicates (open OR closed) of this issue in the repo.
- [x] I have read the contribution guidelines given at https://github.com/TeamNewPipe/NewPipe/blob/HEAD/.github/CONTRIBUTING.md.
- [x] This issue contains only one feature request. I will open one issue for every feature I want to request.
Describe the feature you want
I noticed that some videos on this channel are played with non-English audio track: https://www.youtube.com/channel/UCBSs9x2KzSLhyyA9IKyt4YA (look for the "Multi-Audio Clip" ones)
The same video is played in English on mobile YouTube website. Changing the UI language (Settings -> Account -> Language) also changes the audio track language.
Is your feature request related to a problem? Please describe it
Yes. Some videos get played in a language I don't understand, while mobile YouTube web plays them in English by default (at least on my phone, tested on Firefox Klar/Focus with cookies cleared).
Additional context
See the channel link above.
How will you/everyone benefit from this feature?
This feature will let our users select which language they prefer and watch a video with corresponding audio track. It will also fix the problem I described above.
Mod edit for search keywords: different other
@TiA4f8R sorry, I used FastHub for creating this issue and it does not support templates. Fixed. Please reopen.
This is where we can keep the multiple audio tracks. Orange box for caption drop-down menu and green box for audio track drop-down menu.
All or selected audio tracks should be available for download as well.
I think this would require an extractor change as well. Here are some example videos that I was able to find: Would You Sit In Snakes For $10,000? by Mr Beast Assasin's Creed Valhalla: Story Trailer by Ubisoft North America
The setting is found here on the youtube desktop site:
Any updates to this? Watching MrBeast in Spanish is really annoying and I have to use YouTube app for such videos.
Yes, an initial extractor support has been added. Support of getting whether an audio is descriptive needs to be added in the extractor, and the corresponding app part needs to be done.
found another example of a video that exhibits this issue: https://www.youtube.com/watch?v=ZCbg52OUlyk
That youtuber, ThioJoe, recently made a tool (his announcement of the tool is probably no more than 5 or so videos before the one linked above) that allows other creators to create and upload multiple audio tracks to support other languages using AI. I also have heard that another major youtuber, LinusTechTips, has been exploring a way to automate adding more language tracks to their videos too.
Is there a list of what is needed to help support this feature? i may be willing to chip in some contributions, but as someone brand-new to the codebase, I may need a fair bit of guidance
same problem with the hacksmith that started to put spanish synchronization that gets selected nstead of english (preferred audio set to german): https://m.youtube.com/watch?v=xd8dKY6Ozrg
Running into this on multiple videos of Journey to the Microcosmos. I'm also noticing titles in different languages per 7-ish videos in Trending, could that be part of this as well?
The "Default content language" setting should probably be used for selecting the track it uses even without a per-video selector for now. Could be a quicker way for making NewPipe useable again if the interface design decision is blocking it. Or what is the issue here?
Brutal that this isn't fixed yet. Single handedly ruins this ap, searching for alternatives until they fix this. What good is a video watching app that plays videos I can't understand.. Give me the adds back
The "Default content language" setting should probably be used for selecting the track it uses
This should be indeed used for the default audio language selected.
Or what is the issue here?
The issue is that multiple audio tracks were not supported on the extractor. An initial implementation has been added (see TeamNewPipe/NewPipeExtractor#973), but it makes getting languages objects for clients a bit hard, as you need, for each audio stream, to parse the audio track ID and/or the label, and then convert it into a language object (Java's Locale in our case).
A way to detect whether an audio track is descriptive should be also added, as descriptive audios cannot be distinguished from original audios with this initial implementation.
You can find my implementation about what I talked above here (which also adds language detection of MediaCCC audio single language streams): https://github.com/AudricV/NewPipeExtractor/tree/audio-streams-descriptive-and-locale-properties
Once these missing points are addressed, the app implementation of this feature needs to be done (in the player and in the downloader).
not to be disrespectful, but how is this a feature request? mobile youtube web (which unlike desktop website doesn't have the track selector) only plays non-original audio when there is a track corresponding to the user's native language, while newpipe happily plays non-original audio which is in an arbitrary language that the user is statistically likely to not understand, making it impossible to watch certain videos with newpipe... when the extractor breaks with some youtube change, it usually takes less than a year before the ability to use newpipe for watching youtube videos is restored... this is the same level of criticality (inability to watch youtube videos), only it affects a subset of videos rather than every video (and technically there is a small amount of people who happen to know whatever language is arbitrarily selected) - yet fixing extractor breakage (through no failure of the newpipe team) is not treated as a feature request... is there some workaround available that I'm not aware of, making it less of a priority to fix this asap?
making it less of a priority to fix this asap?
Could it be it a resource issue? Is newpipe in need of more contributors with knowledge of how the extractor works? funding for existing devs? People willing to test or document the fix so it can get through the process faster?
In what ways can newpipe users who are passionate enough to find this issue and add their comments redirect their energy in a way that helps solve this issue?
Is newpipe in need of more contributors with knowledge of how the extractor works?
If you can help out here it would be fantastic.
how is this a feature request?
It's a feature added to YT that Newpipe hasn't caught up to yet.
If you can help out here it would be fantastic.
unfortunately I'm not personally familiar with the codebase. Is this something that's likely to be able to be picked up by a new contributor (or is there a list of things that need doing)?
Nope. But if you have the ability to kidnap some unwary devs and threaten them into contributing, that works too.
Theo Joe make an auto dubber and it's catching on (https://www.youtube.com/watch?v=uaHmtJrZRdM) so this issue is going to be more important
Is this relevant? At 2:45 https://www.youtube.com/watch?v=cas5CVIrjA0
... you've been putting off fixing something that makes certain channels unusable for TWO YEARS???
Does this code from the LibreTube repo help?
private fun getAudioStreamGroups(audioStreams: List<PipedStream>?): Map<String?, List<PipedStream>> {
return audioStreams.orEmpty()
.groupBy { it.audioTrackName }
}
override fun onAudioStreamClicked() {
val audioGroups = getAudioStreamGroups(streams.audioStreams)
val audioLanguages = audioGroups.map { it.key ?: getString(R.string.default_audio_track) }
BaseBottomSheet()
.setSimpleItems(audioLanguages) { index ->
val audioStreams = audioGroups.values.elementAt(index)
val lang = audioStreams.firstOrNull()?.audioTrackId?.substring(0, 2)
trackSelector.updateParameters {
setPreferredAudioLanguage(lang)
}
}
.show(childFragmentManager)
}
https://github.com/libre-tube/LibreTube/blob/master/app/src/main/java/com/github/libretube/ui/fragments/PlayerFragment.kt
by my crude understanding, it seems like there may be some features missing in the underlying extractor that provides the raw data (such as streams.audioStreams in your example code) that would be needed to support this. I took a brief look at some of the linked issues and it seems partly there tho, the bulk of the remaining work seems to be UI/UX changes.
Your code does raise an interesting question though, it looks like that code is using string manipulation to detect the language. Where do the names of those tracks come from? are they defined by the uploader? or does youtube set them to something thats (mostly) consistent across the majority of youtube videos on the platform?
Does this code from the LibreTube repo help?
No, not at all, as LibreTube uses a different way to get audio language tags that we will be able to use, thanks to my changes and data is different between LibreTube which uses Piped's API which uses the extractor one and NewPipe which uses directly the extractor API.
Your code does raise an interesting question though, it looks like that code is using string manipulation to detect the language. Where do the names of those tracks come from? are they defined by the uploader? or does youtube set them to something thats (mostly) consistent across the majority of youtube videos on the platform?
That's what I said in my previous comment, we have to parse an audio track ID to get an audio language code (we could also use the stream URL). All of this is defined by YouTube.
To keep you updated, TeamNewPipe/NewPipeExtractor#1026, which contains the changes I talked above, has been opened and merged. There is only one mistake that I made on the DASH manifest role attribute for descriptive audios I need to fix and then the app implementation could start.
A similar implementation to the one of the subtitles could be the way to implement the audio switch UI, but I am not sure, as services such as MediaCCC use progressive media and ExoPlayer would probably have no information about what language an audio track would have.
For audio selection, the audio selection needs to be changed to return probably a list of audio streams with different languages and a specific quality (in ListHelper).
An external audio player stream dialog needs also to be implemented, which would allow language selection and probably quality selection like the video dialog one.
Note that @Theta-Dev showed on the NewPipe subreddit a preview of this feature, so they may share their implementation insights and/or their ideas.
since this has been in demand for almost two years, would it be acceptable to implement the NewPipe frontend changes in 2 stages? i.e. stage 1 being a basic-but-imperfect implementation that just picks an audio track that matches the language setting device to reduce the number of disrupted viewing experiences, and stage 2 being the full UI implementation that lets the user pick their own language track if they want.
on one hand i see some benefit in doing this in two passes to deal with the problem for the majority of users, but also doing it this way seems like it might take a lot of the pressure off of this issue and could cause phase 2 to be de-prioritized in comparison to more important issues (i.e. "theres nothing more permanent than a temporary fix").
