Support for auto-translated YouTube closed-captions

[trizen]

Checklist

• [X] I made sure that there are no existing issues - open or closed - which I could contribute my information to.
• [X] I'm aware that this is a request for NewPipe itself and that requests for adding a new service need to be made at NewPipeExtractor.
• [X] I have taken the time to fill in all the required details. I understand that the feature request will be dismissed otherwise.
• [X] This issue contains only one feature request.
• [X] I have read and understood the contribution guidelines.

Feature description

Support for auto-translated YouTube closed-captions.

This can be achieved by adding the tlang=LANG URI parameter to the provided closed-captions URLs (including the auto-generated ones), where LANG is the language code in which we want to translate the closed-caption into.

The list of accepted language codes, is:

      af am ar az be bg bn bs ca ceb co cs cy da de el en eo es et eu fa fi fil
      fr fy ga gd gl gu ha haw hi hmn hr ht hu hy id ig is it iw ja jv ka kk km
      kn ko ku ky la lb lo lt lv mg mi mk ml mn mr ms mt my ne nl no ny or pa pl
      ps pt ro ru rw sd si sk sl sm sn so sq sr st su sv sw ta te tg th tk tr tt
      ug uk ur uz vi xh yi yo zh-Hans zh-Hant zu

Example (auto-generated closed-caption in English):

https://www.youtube.com/api/timedtext?v=HiMaSrXaFz0&asr_langs=de,en,es,fr,id,it,ja,ko,nl,pt,ru,tr,vi&caps=asr&exp=xctw&xoaf=5&hl=en&ip=0.0.0.0&ipbits=0&expire=1647400208&sparams=ip,ipbits,expire,v,asr_langs,caps,exp,xoaf&signature=1D31C0E268469B8454678AE346B6AE2B040B02EB.7924D7A5830FB2DDFB22F1CE281AB155BD82ACEA&key=yt8&kind=asr&lang=en&fmt=srv1

Closed-caption auto-translated into Romanian:

https://www.youtube.com/api/timedtext?v=HiMaSrXaFz0&asr_langs=de,en,es,fr,id,it,ja,ko,nl,pt,ru,tr,vi&caps=asr&exp=xctw&xoaf=5&hl=en&ip=0.0.0.0&ipbits=0&expire=1647400208&sparams=ip,ipbits,expire,v,asr_langs,caps,exp,xoaf&signature=1D31C0E268469B8454678AE346B6AE2B040B02EB.7924D7A5830FB2DDFB22F1CE281AB155BD82ACEA&key=yt8&kind=asr&lang=en&fmt=srv1&tlang=ro

This feature can be added as a settings option, where the user inserts the language code that he wants the closed-captions to be translated into, and the closed-caption menu would display an option for selecting the auto-translated caption(s) with the language-code(s) provided by the user.

Why do you want this feature?

This feature is useful for people that do not understand the English language. As most YouTube videos have auto-generated captions, providing auto-translated captions allows people that do not understand the language spoken into the video to be able to understand (to some extent) what is being said in the video.

Additional information

No response

[SameenAhnaf]

This feature can be added as a settings option, where the user inserts the language code that he wants the closed-captions to be translated into

I don't think any option is required to be kept in settings. Add a secondary menu to Auto-translated > in captions menu. Previously selected output languages should be on top. Other languages should be kept in the bottom alphabetically.

[trizen]

As there don't seem to be any plans for adding this feature in the near future, I present below a patch that enables support for auto-translated subtitles for a specific language-code specified at compile-time.

Maybe someone can extend this patch and allow the language-code to be changed during the runtime (in the settings).

Patch for NewPipe:

diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties
index 4ed3bde..92f06b5 100644
--- a/gradle/wrapper/gradle-wrapper.properties
+++ b/gradle/wrapper/gradle-wrapper.properties
@@ -1,6 +1,5 @@
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
 distributionUrl=https\://services.gradle.org/distributions/gradle-7.4.2-all.zip
-distributionSha256Sum=e6d864e3b5bc05cc62041842b306383fc1fefcec359e70cebb1d470a6094ca82
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists
diff --git a/settings.gradle b/settings.gradle
index 0338fde..a0aed95 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -4,8 +4,8 @@ include ':app'
 // We assume, that NewPipe and NewPipe Extractor have the same parent directory.
 // If this is not the case, please change the path in includeBuild().
 
-//includeBuild('../NewPipeExtractor') {
-//    dependencySubstitution {
-//        substitute module('com.github.TeamNewPipe:NewPipeExtractor') using project(':extractor')
-//    }
-//}
+includeBuild('../NewPipeExtractor') {
+    dependencySubstitution {
+        substitute module('com.github.TeamNewPipe:NewPipeExtractor') using project(':extractor')
+    }
+}

Patch for NewPipeExtractor:

diff --git a/extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java b/extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
index a5cb04b..bb4f21b 100644
--- a/extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
+++ b/extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
@@ -657,6 +657,15 @@ public class YoutubeStreamExtractor extends StreamExtractor {
                         .setLanguageCode(languageCode)
                         .setAutoGenerated(isAutoGenerated)
                         .build());
+
+                if (i == 0) {
+                    subtitlesToReturn.add(new SubtitlesStream.Builder()
+                        .setContent(cleanUrl + "&fmt=" + format.getSuffix() + "&tlang=ro", true)
+                        .setMediaFormat(format)
+                        .setLanguageCode("ro")
+                        .setAutoGenerated(isAutoGenerated)
+                        .build());
+                }
             }
         }
 

(replace the language-code "ro" with a language-code of your choice)

Now a debug app can be built with Android Studio. Here's the result:

[crti]

It would be wonderful if auto-translated captions (preference by setup in settings) could be selected trough GUI instead of only the language in the video.

[neofeo]

Would love to have this feature!

[TransZAllen]

Hi, I noticed that the YouTube subtitle auto-translate feature is no longer included in NewPipe, and I was wondering why that is. I’ve seen some apps (forked from NewPipe) that still have this feature, and it’s really useful for users who don’t speak the video's original language. I think this feature could attract more users. Any chance this feature could be added back in the future??? Thanks!

[TransZAllen]

Hi team

I recently tried to build the feature/youtube-auto-translated-captions branch (https://github.com/TeamNewPipe/NewPipeExtractor/tree/feature/youtube-auto-translated-captions), because I was curious about the old auto-translated subtitles feature.

Unfortunately, the branch is quite outdated and failed to build due to several missing classes and old dependencies.
So I eventually gave up trying to install it on my phone.

Just to be clear, I’m not planning to change or maintain that old branch — I just wanted to try it out and see how the auto-translation feature worked.

[TransZAllen]

Here’s a small idea 💡 for the auto-translated subtitles:

Step 1: Add/restore the auto-translation option for YouTube subtitles.
Step 2: For better translation quality, prefer creator-uploaded (not auto-generated) subtitles as the source, since they’re usually more accurate.
Step 3: Add bilingual subtitle display — showing both the original and translated lines together.

If these steps could be realized, I think NewPipe’s subtitle experience would become even more attractive and fun to use 🌍 😃