danielmiessler
[Bug]: Malformed YT Helper function output
What happened?
When using the YT helper function as such:
yt "<video url>" |
the output contains "\u0026#39;" instead of an apostrophe. I was able to workaround with:
yt "<video url>" | %{$_.replace('\u0026#39;',"'")}
Version check
- [x] Yes I was.
Relevant log output
Relevant screenshots (optional)
No response
The output of pulling YoutTube transcripts can be altered/affected by the various URLs that all point to the same video. For a complete breakdown on this issue, you may refer to unofficial video documentation on this specific topic here:
https://youtu.be/6IxC-63Owx4
somewhere around 08:45 should start to zero in to your problem
If, after learning how and where to grab clean YouTube URLs for the Fabric tool, you are still having problems then post back here with your input, output, etc. Don't forget to use the --dry-run flag to discover exactly what is being sent by the Fabric CLI tool.
fabric -y "https://youtube.com/watch?v=uXs-zPc63kM" --stream --pattern extract_wisdom
don't give any output
fabric -y "https://youtube.com/watch?v=uXs-zPc63kM" dont give any output
fabric -y "https://youtube.com/watch?v=uXs-zPc63kM" --comments list all comments
fabric -y "https://youtube.com/watch?v=uXs-zPc63kM" --stream dont give any output
fabric -y "https://youtube.com/watch?v=uXs-zPc63kM" --stream --pattern extract_wisdom I'm sorry, but it seems there is no content provided for me to analyze. Could you please provide the text or information you'd like me to process?
fabric --version v1.4.166
Let's try to zero in on where things are going wrong for you.
If the command you tried: fabric -y "https://youtube.com/watch?v=uXs-zPc63kM" ...does not give any output, the next most obvious thing to check is if your YouTube API keys were entered correctly.
Did you go on to developers.google.com and find your way to creating a YouTube API key? And then enter that key (a long string of random characters) into the YouTube Plugin inside the Fabric Setup Menu?
Please be patient with me, if i ask questions about things you know you've already done. I don't know that you've done them, and I have to ask in order to trace the problem.
Let's try to zero in on where things are going wrong for you.
If the command you tried: fabric -y "https://youtube.com/watch?v=uXs-zPc63kM" ...does not give any output, the next most obvious thing to check is if your YouTube API keys were entered correctly.
Did you go on to developers.google.com and find your way to creating a YouTube API key? And then enter that key (a long string of random characters) into the YouTube Plugin inside the Fabric Setup Menu?
Please be patient with me, if i ask questions about things you know you've already done. I don't know that you've done them, and I have to ask in order to trace the problem.
Hi thank you for answer. Yes I placed YouTube API keys. I think getting youtube comments listed might prove that API key is provided. Additionally please find screen from API console - I see numbers increasing everytime I'm placing fabric -y "https://youtube.com/watch?v=uXs-zPc63kM" --comments
I believe you are correct. If your youtube keys are incorrect or missing, the output would look like this:
fabric -y https://youtu.be/6IxC-63Owx4 --comments YouTube is not configured, please run the setup procedure
I need to try and recreate your situation/scenario in order to figure out what is going wrong for you.
fabric -y https://youtube.com/watch?v=uXs-zPc63kM #provides no output for you fabric -y https://youtube.com/watch?v=uXs-zPc63kM --comments #outputs the comments for you fabric -y https://youtube.com/watch?v=uXs-zPc63kM --metadata #does this output the videos metadata for you??
If comments and metadata are provided via your youtube keys to the fabric cli tool, but the transcript is not, I have to suspect that the fabric tool is not malfunctioning, and that the transcript is not being sent from the api call to youtube. Let me look into this, to see if others experience this when pulling data from their keys (by means of things other than the fabric tool).
From a cursory internet search:
YouTube API Rate Limit for Transcripts There is a rate limit on pulling transcripts with YouTube API keys. The daily quota for the YouTube Data API is set at 1,000,000 units, but for most projects, the default quota allocation is 10,000 units per day. Each operation you perform consumes a specific number of units, such as 1 unit for a read operation, 50 units for a write operation, and 1600 units for a video upload. If you exceed your quota, you can request additional quota by completing the Quota extension request form for YouTube API Services. It's important to be mindful of these limits to prevent service interruptions.
Additionally, the YouTube Transcript API also has usage restrictions and rate limits to prevent abuse and ensure fair usage. Making too many requests in a short period could result in temporary blocks or restrictions.
To manage your API usage effectively, you can implement rate limiting in your code and familiarize yourself with the quota costs of each API method
So, the next step is to understand the timeline of your problem. Have you been pulling transcripts successfully for a long time, and suddenly you no longer can? Has it been a day to two that you are unable to pull transcripts, or much longer? If you have been blocked from pulling transcripts, your lower quota might still allow you to pull comments, thus obfuscating the problem...
NOTE: I have found individuals on Stack Overflow that have been blocked by YouTube from pulling transcripts. They are trying to reason through the 10K daily limit, and reconciling that with the 2 rpm limit on pulling them. A normal transaction is probably less than 1 transcript request per minute, and blocks are thrown up when this is drastically increased. The API key block is not permanent, it appears. This happens in an effort to stop bots and people with nefarious intent.
After additional testing I found that this:
fabric -y '<video url>'
Produces clean text but this:
yt '<video url>'
Produces text with every apostrophe replaced with '\u0026#39;'.
I believe you are correct. If your youtube keys are incorrect or missing, the output would look like this:
fabric -y https://youtu.be/6IxC-63Owx4 --comments YouTube is not configured, please run the setup procedure
I need to try and recreate your situation/scenario in order to figure out what is going wrong for you.
fabric -y https://youtube.com/watch?v=uXs-zPc63kM #provides no output for you fabric -y https://youtube.com/watch?v=uXs-zPc63kM --comments #outputs the comments for you fabric -y https://youtube.com/watch?v=uXs-zPc63kM --metadata #does this output the videos metadata for you??
If comments and metadata are provided via your youtube keys to the fabric cli tool, but the transcript is not, I have to suspect that the fabric tool is not malfunctioning, and that the transcript is not being sent from the api call to youtube. Let me look into this, to see if others experience this when pulling data from their keys (by means of things other than the fabric tool).
From a cursory internet search:
YouTube API Rate Limit for Transcripts There is a rate limit on pulling transcripts with YouTube API keys. The daily quota for the YouTube Data API is set at 1,000,000 units, but for most projects, the default quota allocation is 10,000 units per day. Each operation you perform consumes a specific number of units, such as 1 unit for a read operation, 50 units for a write operation, and 1600 units for a video upload. If you exceed your quota, you can request additional quota by completing the Quota extension request form for YouTube API Services. It's important to be mindful of these limits to prevent service interruptions.
Additionally, the YouTube Transcript API also has usage restrictions and rate limits to prevent abuse and ensure fair usage. Making too many requests in a short period could result in temporary blocks or restrictions.
To manage your API usage effectively, you can implement rate limiting in your code and familiarize yourself with the quota costs of each API method
So, the next step is to understand the timeline of your problem. Have you been pulling transcripts successfully for a long time, and suddenly you no longer can? Has it been a day to two that you are unable to pull transcripts, or much longer? If you have been blocked from pulling transcripts, your lower quota might still allow you to pull comments, thus obfuscating the problem...
NOTE: I have found individuals on Stack Overflow that have been blocked by YouTube from pulling transcripts. They are trying to reason through the 10K daily limit, and reconciling that with the 2 rpm limit on pulling them. A normal transaction is probably less than 1 transcript request per minute, and blocks are thrown up when this is drastically increased. The API key block is not permanent, it appears. This happens in an effort to stop bots and people with nefarious intent.
No I never before start pooling transcripts. I installed fabric today and would like to try it. I tryied manually few times by using commands from installation example. PI Rate Limit for Transcripts should be not a case here.
Yes https://youtube.com/watch?v=uXs-zPc63kM --metadata does output as below
POSSIBLE ROOT CAUSES:
Scenario A: The Video Has No Transcript Available Not all YouTube videos have auto-generated or manually uploaded transcripts. You should check if the video has subtitles by:
Going to the video on YouTube and clicking the CC (Closed Captions) button.
Checking if a transcript is available by clicking the three dots below the video → Show transcript.
If no transcript is available on YouTube, fabric -y cannot retrieve it.
Have you tried videos from other channels with other videos?
Scenario B: Using --transcript-with-timestamps Instead Some videos may return transcripts only when requesting timestamps.
fabric -y https://youtube.com/watch?v=uXs-zPc63kM --transcript-with-timestamps
Note: I will look into this deeper. I truly suspect you have been subject to mis-applied rate limiting, but only time will tell if that is true.
Looks like video has transcript. Unfortunatelly fabric -y https://youtube.com/watch?v=uXs-zPc63kM --transcript-with-timestamps also not giving transcript. Are there any other ways how to get transcription for videos from yt?
There are other ways. If you are on linux (anything Debian based, for apt) you can: sudo apt install yt-dlp
I have already tried that for the video you are trying to pull a transcript from, and it says FORBIDDEN but the yt-dlp command line tool does allow me to pull comments.
If you are familiar with python, you can build a tiny tool with the youtube transcript api library:
After setting up a virtual environment:
pip install youtube-transcript-api
And then write a simple script. I don't know your background to know what to suggest. Turn key tools? yt-dlp is probably the way to go. It has a book's worth of options, so I'm not positive I'm using it correctly yet.
Have you tried other videos, and still don't get a transcript? Or is it just this one guy's channel and/or videos that don't provide you a transcript?
Hi that was great discussion. The problem is on OVH servers -I have private IP there which should not be used by anyone but I guess this address even its dedicated to me its in OVH pool which is restricted.
When I use the same API key, they same fabric version on VM in home everything works for me perfectly fine. I hope this will help other with troubleshooting. It might be worth to consider for fabric some kind verbose mode which will give more detailed informations that for example yt block IP.
Thank you for your help!
Glad you figured it out! A verbose or debugging mode would help immensely, I agree. As developed as this Fabric AI project is, we should remember that it is approximately 1 year old. That's still very new. I'm sure the fabric developers will integrate all of these things, as time allows. Thanks for your patience!
i am trying to summarize youtube vid at home, an i am still facing the same issue. comments & metadata work. transcriptions don't. i have the usual home router configurations. i used it from a server in Norway which had the same issue. no transcription for initial 2 3 time, then it work for once. and then 429 Too Many Requests. is that the limit? just one?
Can you send the YouTube link (or does it happen with all of them?) and I'll see if I can replicate it.
@nileshevrywhr did you read the discussion above?
https://github.com/danielmiessler/fabric/issues/1395#issuecomment-2764289736
v1.4.196
For me, it seems intermittent. I ran yt https://youtu.be/GoMeXbsRPwA?si=I341MXx5RITmhmlO 10 times, twice came back with transcript, the other eight times as "transcript not available. (EOF)". So either Google API is throttling or yt is timing out maybe?
Hmmm... Maybe YouTube is throttling more than they used to. I'm seeing such errors more often for myself too.
Can you send the YouTube link (or does it happen with all of them?) and I'll see if I can replicate it.
@nileshevrywhr did you read the discussion above?
I used the example video that's in README. i have the same version v1.4.196 as @leerogers.
@ksylvan I went through the discussion 3 times. it's quite enlightening. then went through the documentation to check the limits and API console to track my usage and i'm barely using it. I even timed the gap in my requests.
now, i suppose i'll use the APIs Explorer in the documentation, to see what's happening.
@nileshevrywhr I don't know. 🤔
I do know that no changes in those parts of the code have happened in a long, long time. I'll take another deeper look.
Can you send the YouTube link (or does it happen with all of them?) and I'll see if I can replicate it.
For me, it can be any youtube link
@leerogers @nileshevrywhr @MichalPoterek @b1naryagent Please pull the latest and try again.
@leerogers @nileshevrywhr @MichalPoterek @b1naryagent Please pull the latest and try again.
Hi @ksylvan , thanks for working on this. I installed v1.4.200, installed yt-dlp and tested. Both commands come with the same error with several different YouTube URLs. Also tried a new YouTube API key.
yt https://www.youtube.com/watch?v=R6fudpTKdUU fabric -y https://www.youtube.com/watch?v=R6fudpTKdUU
yt-dlp failed: exit status 1, stderr:
ERROR: Did not get any data blocks
Tried on a VPN and got this error:
yt https://youtu.be/AS5wspByLzw?si=qUNcnfeY8ifMLVFg yt-dlp failed: exit status 1, stderr: ERROR: [youtube] AS5wspByLzw: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
Any ideas? Any additional configuration needed? I see yt-dlp has options to auth or use cookies need-be but we shouldn't need that, right?
No additional configuration needed.
$ fabric -y 'https://www.youtube.com/watch?v=R6fudpTKdUU' --transcript-with-timestamps| head -10
[00:00:00] What time is it? 4:20.
[00:00:02] What time is it? 4:20.
[00:00:02] What time is it? 4:20.
[00:00:02] What time? 4:20. And what do we want?
[00:00:06] What time? 4:20. And what do we want?
[00:00:06] What time? 4:20. And what do we want?
[00:00:06] What? Say it again. Yeah.
[00:00:15] And to answer the cries, the yells, we
[00:00:18] And to answer the cries, the yells, we
[00:00:18] And to answer the cries, the yells, we
What version is your yt-dlp?
yt-dlp --version
2025.06.09
And fabric version:
fabric --version
v1.4.200
yt-dlp --version
Sorry, looks like an older version of yt-dlp came from Debian apt get. Let me see how to install it manually.
lee@Zenbook:~/.config/fabric$ yt-dlp --version 2024.04.09 lee@Zenbook:~/.config/fabric$ fabric --version v1.4.200
Sorry, looks like an older version of yt-dlp came from Debian apt get. Let me see how to install it manually.
You could probably just grab the binary: https://github.com/yt-dlp/yt-dlp#release-files
I'm lucky because "brew" updated it (the latest release is only 2 days old)
OK, bingo. Installed latest version and transcripts ran 5 times in a row.
lee@Zenbook:~$ yt-dlp --version 2025.06.09
lee@Zenbook:~$ fabric -y 'https://www.youtube.com/watch?v=R6fudpTKdUU' --transcript-with-timestamps| head -10 [00:00:00] What time is it? 4:20. [00:00:02] What time is it? 4:20. [00:00:02] What time is it? 4:20. [00:00:02] What time? 4:20. And what do we want? [00:00:06] What time? 4:20. And what do we want? [00:00:06] What time? 4:20. And what do we want? [00:00:06] What? Say it again. Yeah. [00:00:15] And to answer the cries, the yells, we [00:00:18] And to answer the cries, the yells, we [00:00:18] And to answer the cries, the yells, we
Thank you @ksylvan , you are a savior.
Now we can all go back to not watching YouTube and pretending like we did. 😁