Preventing Error 429: Too Many Requests

[nicolasdeory]

I'm using ytdl-core from a browser, using browserify. Cross-domain requests are not allowed, so I'm using CORS Anywhere proxy to bypass CORS Header related errors.

After a few requests, I'm getting error 429s from YouTube, saying that I may be making too many requests (not really, one or two per minute at the most, downloading a webm stream). The method crashes afterward and won't do any additional requests.

Google seems to block the entire origin domain, so it won't work for anyone with the same origin header (I haven't verified yet).

There are multiple things wrong:

• How can I prevent this? I'd like to have some sort of rate throttling so it doesn't incur into 429 errors. It renders ytdl completely unusable for a long time (no exact amount, last time it was more than an hour)
• No error is thrown by ytdl whatsoever. This is not good because there is no way to handle the error. The callback never gets invoked, and error handlers are never called.

Code example:

const ytdl = require('ytdl-core');
const concat = require('concat-stream');
 var concatStream = concat((arrayBuffer) =>
        {
            console.log('file downloaded');
           // Processing of video blob goes here...
        });

        var ytstream = ytdl(YOUTUBE_VIDEOURL,
            {
                requestOptions: {
                    transform: (parsed) =>
                    {  
                        var parsedHeaders = parsed.headers;
                        return {
                            host: 'cors-anywhere.herokuapp.com/', 
                            path: "http://youtube.com" + parsed.path,
                            maxRedirects: 10,
                            headers: parsedHeaders
                        }
                    },
                },
            });

        ytstream.on('error', function(e)
        {
           // Never gets called
        })
        .pipe(concatStream);

[fent]

• How can I prevent this? I'd like to have some sort of rate throttling so it doesn't incur into 429 errors. It renders ytdl completely unusable for a long time (no exact amount, last time it was more than an hour)

I'm going to be adding throttling so that it doesn't make requests too fast to prevent these errors

• No error is thrown by ytdl whatsoever. This is not good because there is no way to handle the error. The callback never gets invoked, and error handlers are never called.

will be fixed soon

[akanellis]

I am a fairly new JS developer I come from old school C/C++ background and I am seriously really confused with this.

I am trying to get ytdl-core to pass cookies to miniget because this is a solution for the 429 error. I have tested using cookies with youtube-dl and it works perfectly but I want to make it work with ytdl-core because that other project is using deprecated libraries and I don't feel good with that.

const ytdl = require('ytdl-core');
const cookiefile = require('cookiefile');

const cookiemap = new cookiefile.CookieMap('cookies.txt');
const cookies = cookiemap.toRequestHeader().replace ('Cookie: ','');

stream = ytdl(URL, {
    quality: 'highestaudio',
    highWaterMark: 1024 * 1024 * 1024,
    requestOptions: {
        headers: {
            'Cookie': cookies
        }
    }
});

cookies.txt is a cookies export from chrome. I have also modified miniget's index.js so that it prints out variable "parsed" on line 126 just so that I can see what I'm doing. When I run the above code, I get the following output for parsed:

Url {
  protocol: 'https:',
  slashes: true,
  auth: null,
  host: 'www.youtube.com',
  port: null,
  hostname: 'www.youtube.com',
  hash: null,
  search: '?v=Rbm6GXllBiw&hl=en&bpctr=1590461824&pbj=1',
  query: 'v=Rbm6GXllBiw&hl=en&bpctr=1590461824&pbj=1',
  pathname: '/watch',
  path: '/watch?v=Rbm6GXllBiw&hl=en&bpctr=1590461824&pbj=1',
  href: 'https://www.youtube.com/watch?v=Rbm6GXllBiw&hl=en&bpctr=1590461824&pbj=1',
  maxRedirects: 2,
  maxRetries: 2,
  maxReconnects: 0,
  backoff: { inc: 100, max: 10000 },
  headers: {
    Cookie: 'my-cookies-here',
    'x-youtube-client-name': '1',
    'x-youtube-client-version': '2.20191008.04.01'
  }
}

Under headers, "cookie" is not inside quotes just like the rest of the headers and I guess it's natural that I get the following error:

(node:7309) UnhandledPromiseRejectionWarning: Error: Error parsing info: JSON.parse(...).reduce is not a function
    at exports.getBasicInfo (/home/user/project/node_modules/ytdl-core/lib/info.js:43:11)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async Object.exports.<computed> [as getBasicInfo] (/home/user/project/node_modules/ytdl-core/lib/info.js:297:18)
    at async exports.getFullInfo (/home/user/project/node_modules/ytdl-core/lib/info.js:188:14)
    at async Object.exports.<computed> [as getFullInfo] (/home/user/project/node_modules/ytdl-core/lib/info.js:297:18)

But when I use any other header name (for example 'custom-header', it gets inserted correctly like this:

Url {
  protocol: 'https:',
  slashes: true,
  auth: null,
  host: 'www.youtube.com',
  port: null,
  hostname: 'www.youtube.com',
  hash: null,
  search: '?v=TnAgc1kgvLc&hl=en&bpctr=1590462647&pbj=1',
  query: 'v=TnAgc1kgvLc&hl=en&bpctr=1590462647&pbj=1',
  pathname: '/watch',
  path: '/watch?v=TnAgc1kgvLc&hl=en&bpctr=1590462647&pbj=1',
  href: 'https://www.youtube.com/watch?v=TnAgc1kgvLc&hl=en&bpctr=1590462647&pbj=1',
  maxRedirects: 2,
  maxRetries: 2,
  maxReconnects: 0,
  backoff: { inc: 100, max: 10000 },
  headers: {
    'custom-header': 'my-cookies-here',
    'x-youtube-client-name': '1',
    'x-youtube-client-version': '2.20191008.04.01'
  }
}

The above does not produce a JSON.parse error, it produces the usual Error: Status code: 429 which means I guess that the header was valid and went through the request.

So whichever name I use for header name is valid except for Cookie. Why is that? I've been trying to figure this out for the past 6 hours. I apologize in advance if my question is silly. My JS background is very limited.

[fent]

the way that console.log prints objects, it will put quotes around attribute names that contain a non-alphanumeric character, it does not affect the actual internal attribute name used.

as for why it's not working and the error you're getting, try lowercasing the header name to "cookie".

if that doesn't work, it's likely that having that cookie there prevents the endpoint ytdl-core uses internally to work. you could try only setting the cookie during the download. something like

let info = await ytdl.getInfo(id);
ytdl.downloadFromInfo(info, { requestOptions: { Cookie: cookie } });

[akanellis]

Thank you for the speedy answer. I'm afraid this doesn't work.. I still get the 429 error but now I get it from the getInfo() function. I need to set cookie for every request that I send to youtube and I can't find an example code anywhere online about setting cookies with ytdl-core that works :(

Maybe you could add an example for cookies in the example folder than you have?

If you are not throttled by youtube with error 429 you can recreate the scenario trying to download a private video. Only when using cookies you will be able to get it.

[harshmandan]

Same here. Getting a bursty stream of requests blocks all videos. Would love to see a correct cookie implementation

[FeX5858]

I have a similar problem with my discord.js bot and ytdl by itself. If i use ytdl "anyurl" --print-url it throws the Error: 429. When i use ytdl-core for my discord bot, the bot joins but has no audio. The player doesnt receive any ytdl data. (buffer length: 0). The error started around 7 to 8 months using the bot on 1 Server.

[NafiAF]

I am also getting this error, it was working fine all this time (since I started using it, 3 months ago) but now I am suddenly getting a 429 error with my Discord Bot. It still happens after small requests even when the Bot did not request any data for around 12 hours.

[FeX5858]

@NafiAF I solved the problem for me. I waited around 4 to 5 days and then it was gone. The error triggeres when ur bot gets too many requests in a small amount of time. Then Google blocks the bot for some days to prevent DDOS. This is what works for me :)

[akanellis]

@NafiAF Personally I have solved this issue by not using ytdl-core anymore. I have moved to youtube-dl because that one has easy support for cookies. When you use cookies so that you are logged into youtube, the error 429 does not occur anymore.

[mahhov]

For what it's worth, @fent and @akanellis 's solution of using cookies worked, at least for now. I just hope my account doesn't get blocked or something.

I navigated to youtube.com in chrome, opened dev console, and copied the cookies from a network request. Then do:

ytdl(<id>, {
            ...
            requestOptions: {
                headers: {
                    'Cookie': '<paste cookie as string>'
                }
            }
        })

[mahhov]

Apparently, even using oauth isn't immune to 429. Using oauth in the request options (e.g. {filter:'audioonly', requestOptions: {headers: {Authorization: 'Bearer ...'}}}gives me ~15-25 videos, before getting another 429. Refreshing the access token (but using the same refresh token and thereby not requiring user re-login) gives me another 15-25 videos. This is just ~1 video per 3-5 minutes. (I'm only 75% this comment is accurate :), I'll update it once I verify it again)

[mahhov]

Confirmed, my above comment is accurate. Refreshing the oath token does resolve the issue. It took 20 downloads to reach http 429. I guess something like this should work then

ytdl(...)
  .on('error', error => {
    if (error.message === 'Status code: 429') {
      // refreh token
      // try again
    }
  })

[eric-mathison]

I'm 429 blocked now too so I've been playing around with cookies. I've narrowed it down to these three: 'SID=xxx; HSID=xxx; SSID=xxxx;'

I've been able to make successful calls after setting these manually. They are all set to expire in July 2022, so hopefully I can get some good use out of them until then.

[WaqasIbrahim]

@eric-mathison Were you able to make requests from a blocked IP?

[eric-mathison]

@eric-mathison Were you able to make requests from a blocked IP?

I couldn't confirm that my IP was blocked, but I kept getting error 429 from my node application running local on my laptop. I waited over 14 hours and it was still receiving this error. Passing those 3 cookies is working. I just finished downloading 52 videos in around 7 minutes time. They are small 75mb and I downloaded them slow, 1 at a time. So far so good.

[WaqasIbrahim]

Does not seem to work on a server that's already blocked.

Edit: Same request works fine in Postman.

[fent]

Does not seem to work on a server that's already blocked.

Edit: Same request works fine in Postman.

it's showing 10 headers in postman, maybe that's helping

[WaqasIbrahim]

@fent Those are standard headers, I generated the curl command using postman.

Update: I disabled IPv6 on the server. Without headers and cookie:

With headers and cookie

Update 2: Did not last long, went back to 429.

[eric-mathison]

Does not seem to work on a server that's already blocked.

Edit: Same request works fine in Postman.

@WaqasIbrahim that doesn't look like an IP block. You are getting blocked by reCaptcha. See if you can curl the get_video_info url directly. YouTube doesn't block the watch url. Only the get info.

[WaqasIbrahim]

@eric-mathison get_video_info seems to be working but watch page URL is used first. get_video_info is a fallback URL.

[eric-mathison]

@eric-mathison get_video_info seems to be working but watch page URL is used first. get_video_info is a fallback URL.

Eh, you're probably right. I was just able to get it working from the get_video_info URL since the watch pages never broke for me.

[Abhay07]

My IP was blocked and i can confirm that. setting proxy or anything didn't work. Had to spin up a new server.

[PhoenixOnARoof]

Services on my host have been having this issue as of recently. In fact, I've been unable to use ytdl at all for the last few days. I have been unsuccessful in using cookies to fix the problem. I'm not sure what to do about it.

[lxcool9243]

Can you add a openvpn or some vpn option as a parameter to use with this. It'll be super helpful.

[jensbrehmen]

I'm 429 blocked now too so I've been playing around with cookies. I've narrowed it down to these three: 'SID=xxx; HSID=xxx; SSID=xxxx;'

I've been able to make successful calls after setting these manually. They are all set to expire in July 2022, so hopefully I can get some good use out of them until then.

This worked for me. I'm getting tons of requests and didn't receive the error anymore.

[lxcool9243]

I'm 429 blocked now too so I've been playing around with cookies. I've narrowed it down to these three: 'SID=xxx; HSID=xxx; SSID=xxxx;' I've been able to make successful calls after setting these manually. They are all set to expire in July 2022, so hopefully I can get some good use out of them until then.

This worked for me. I'm getting tons of requests and didn't receive the error anymore.

Can you describe how to achieve that ?

[jensbrehmen]

For what it's worth, @fent and @akanellis 's solution of using cookies worked, at least for now. I just hope my account doesn't get blocked or something.

I navigated to youtube.com in chrome, opened dev console, and copied the cookies from a network request. Then do:

ytdl(<id>, {
            ...
            requestOptions: {
                headers: {
                    'Cookie': '<paste cookie as string>'
                }
            }
        })

@lxcool9243 The above comment explains how. Just open a request (one that says 'log_event?alt=json...', something like that) at the left when you're in the Network tab on youtube.com in Chrome Dev Tools. Scroll down to the Request Headers and you'll find a string called 'coockie'. This is the one you should copy and paste as in the example above. I did this while I was logged in, but I don't think that has an impact on your outcome.

Use the coockie in every ytdl request, also when you're requesting video info & the hassle will be gone :) It worked perfectly for me.

Hope this helps! Kind regards, Jens.

[lxcool9243]

For what it's worth, @fent and @akanellis 's solution of using cookies worked, at least for now. I just hope my account doesn't get blocked or something. I navigated to youtube.com in chrome, opened dev console, and copied the cookies from a network request. Then do:

ytdl(<id>, {
            ...
            requestOptions: {
                headers: {
                    'Cookie': '<paste cookie as string>'
                }
            }
        })

@lxcool9243 The above comment explains how. Just open a request (one that says 'log_event?alt=json...', something like that) at the left when you're in the Network tab on youtube.com in Chrome Dev Tools. Scroll down to the Request Headers and you'll find a string called 'coockie'. This is the one you should copy and paste as in the example above. I did this while I was logged in, but I don't think that has an impact on your outcome.

Use the coockie in every ytdl request, also when you're requesting video info & the hassle will be gone :) It worked perfectly for me.

Hope this helps! Kind regards, Jens.

Thanks for the explanation @jensbrehmen and for the quick reply.

[aikozijlemans]

This is the issue i have been getting. MinigetError: Status code: 429 at ClientRequest.<anonymous> (C:\Users\HP\Desktop\music\node_modules\ytdl-core-discord\node_modules\miniget\dist\index.js:163:31) at Object.onceWrapper (events.js:422:26) at ClientRequest.emit (events.js:315:20) at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:596:27) at HTTPParser.parserOnHeadersComplete (_http_common.js:119:17) at TLSSocket.socketOnData (_http_client.js:469:22) at TLSSocket.emit (events.js:315:20) at addChunk (_stream_readable.js:295:12) at readableAddChunk (_stream_readable.js:271:9) at TLSSocket.Readable.push (_stream_readable.js:212:10)

[fent]

for those getting this, do you know if the error comes from the getInfo() call, or does it happen after the download has started?