Preventing Error 429: Too Many Requests

[nicolasdeory]

I'm using ytdl-core from a browser, using browserify. Cross-domain requests are not allowed, so I'm using CORS Anywhere proxy to bypass CORS Header related errors.

After a few requests, I'm getting error 429s from YouTube, saying that I may be making too many requests (not really, one or two per minute at the most, downloading a webm stream). The method crashes afterward and won't do any additional requests.

Google seems to block the entire origin domain, so it won't work for anyone with the same origin header (I haven't verified yet).

There are multiple things wrong:

• How can I prevent this? I'd like to have some sort of rate throttling so it doesn't incur into 429 errors. It renders ytdl completely unusable for a long time (no exact amount, last time it was more than an hour)
• No error is thrown by ytdl whatsoever. This is not good because there is no way to handle the error. The callback never gets invoked, and error handlers are never called.

Code example:

const ytdl = require('ytdl-core');
const concat = require('concat-stream');
 var concatStream = concat((arrayBuffer) =>
        {
            console.log('file downloaded');
           // Processing of video blob goes here...
        });

        var ytstream = ytdl(YOUTUBE_VIDEOURL,
            {
                requestOptions: {
                    transform: (parsed) =>
                    {  
                        var parsedHeaders = parsed.headers;
                        return {
                            host: 'cors-anywhere.herokuapp.com/', 
                            path: "http://youtube.com" + parsed.path,
                            maxRedirects: 10,
                            headers: parsedHeaders
                        }
                    },
                },
            });

        ytstream.on('error', function(e)
        {
           // Never gets called
        })
        .pipe(concatStream);

[fent]

• How can I prevent this? I'd like to have some sort of rate throttling so it doesn't incur into 429 errors. It renders ytdl completely unusable for a long time (no exact amount, last time it was more than an hour)

I'm going to be adding throttling so that it doesn't make requests too fast to prevent these errors

• No error is thrown by ytdl whatsoever. This is not good because there is no way to handle the error. The callback never gets invoked, and error handlers are never called.

will be fixed soon

[akanellis]

I am a fairly new JS developer I come from old school C/C++ background and I am seriously really confused with this.

I am trying to get ytdl-core to pass cookies to miniget because this is a solution for the 429 error. I have tested using cookies with youtube-dl and it works perfectly but I want to make it work with ytdl-core because that other project is using deprecated libraries and I don't feel good with that.

const ytdl = require('ytdl-core');
const cookiefile = require('cookiefile');

const cookiemap = new cookiefile.CookieMap('cookies.txt');
const cookies = cookiemap.toRequestHeader().replace ('Cookie: ','');

stream = ytdl(URL, {
    quality: 'highestaudio',
    highWaterMark: 1024 * 1024 * 1024,
    requestOptions: {
        headers: {
            'Cookie': cookies
        }
    }
});

cookies.txt is a cookies export from chrome.
I have also modified miniget's index.js so that it prints out variable "parsed" on line 126 just so that I can see what I'm doing.
When I run the above code, I get the following output for parsed:

Url {
  protocol: 'https:',
  slashes: true,
  auth: null,
  host: 'www.youtube.com',
  port: null,
  hostname: 'www.youtube.com',
  hash: null,
  search: '?v=Rbm6GXllBiw&hl=en&bpctr=1590461824&pbj=1',
  query: 'v=Rbm6GXllBiw&hl=en&bpctr=1590461824&pbj=1',
  pathname: '/watch',
  path: '/watch?v=Rbm6GXllBiw&hl=en&bpctr=1590461824&pbj=1',
  href: 'https://www.youtube.com/watch?v=Rbm6GXllBiw&hl=en&bpctr=1590461824&pbj=1',
  maxRedirects: 2,
  maxRetries: 2,
  maxReconnects: 0,
  backoff: { inc: 100, max: 10000 },
  headers: {
    Cookie: 'my-cookies-here',
    'x-youtube-client-name': '1',
    'x-youtube-client-version': '2.20191008.04.01'
  }
}

Under headers, "cookie" is not inside quotes just like the rest of the headers and I guess it's natural that I get the following error:

(node:7309) UnhandledPromiseRejectionWarning: Error: Error parsing info: JSON.parse(...).reduce is not a function
    at exports.getBasicInfo (/home/user/project/node_modules/ytdl-core/lib/info.js:43:11)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async Object.exports.<computed> [as getBasicInfo] (/home/user/project/node_modules/ytdl-core/lib/info.js:297:18)
    at async exports.getFullInfo (/home/user/project/node_modules/ytdl-core/lib/info.js:188:14)
    at async Object.exports.<computed> [as getFullInfo] (/home/user/project/node_modules/ytdl-core/lib/info.js:297:18)

But when I use any other header name (for example 'custom-header', it gets inserted correctly like this:

Url {
  protocol: 'https:',
  slashes: true,
  auth: null,
  host: 'www.youtube.com',
  port: null,
  hostname: 'www.youtube.com',
  hash: null,
  search: '?v=TnAgc1kgvLc&hl=en&bpctr=1590462647&pbj=1',
  query: 'v=TnAgc1kgvLc&hl=en&bpctr=1590462647&pbj=1',
  pathname: '/watch',
  path: '/watch?v=TnAgc1kgvLc&hl=en&bpctr=1590462647&pbj=1',
  href: 'https://www.youtube.com/watch?v=TnAgc1kgvLc&hl=en&bpctr=1590462647&pbj=1',
  maxRedirects: 2,
  maxRetries: 2,
  maxReconnects: 0,
  backoff: { inc: 100, max: 10000 },
  headers: {
    'custom-header': 'my-cookies-here',
    'x-youtube-client-name': '1',
    'x-youtube-client-version': '2.20191008.04.01'
  }
}

The above does not produce a JSON.parse error, it produces the usual Error: Status code: 429 which means I guess that the header was valid and went through the request.

So whichever name I use for header name is valid except for Cookie. Why is that? I've been trying to figure this out for the past 6 hours. I apologize in advance if my question is silly. My JS background is very limited.

[fent]

the way that console.log prints objects, it will put quotes around attribute names that contain a non-alphanumeric character, it does not affect the actual internal attribute name used.

as for why it's not working and the error you're getting, try lowercasing the header name to "cookie".

if that doesn't work, it's likely that having that cookie there prevents the endpoint ytdl-core uses internally to work. you could try only setting the cookie during the download. something like

let info = await ytdl.getInfo(id);
ytdl.downloadFromInfo(info, { requestOptions: { Cookie: cookie } });

[akanellis]

Thank you for the speedy answer. I'm afraid this doesn't work.. I still get the 429 error but now I get it from the getInfo() function. I need to set cookie for every request that I send to youtube and I can't find an example code anywhere online about setting cookies with ytdl-core that works :(

Maybe you could add an example for cookies in the example folder than you have?

If you are not throttled by youtube with error 429 you can recreate the scenario trying to download a private video. Only when using cookies you will be able to get it.

[harshmandan]

Same here. Getting a bursty stream of requests blocks all videos. Would love to see a correct cookie implementation

[FeX5858]

I have a similar problem with my discord.js bot and ytdl by itself. If i use ytdl "anyurl" --print-url it throws the Error: 429. When i use ytdl-core for my discord bot, the bot joins but has no audio. The player doesnt receive any ytdl data. (buffer length: 0). The error started around 7 to 8 months using the bot on 1 Server.

[NafiAF]

I am also getting this error, it was working fine all this time (since I started using it, 3 months ago) but now I am suddenly getting a 429 error with my Discord Bot. It still happens after small requests even when the Bot did not request any data for around 12 hours.

[FeX5858]

@NafiAF I solved the problem for me. I waited around 4 to 5 days and then it was gone. The error triggeres when ur bot gets too many requests in a small amount of time. Then Google blocks the bot for some days to prevent DDOS. This is what works for me :)

[akanellis]

@NafiAF Personally I have solved this issue by not using ytdl-core anymore. I have moved to youtube-dl because that one has easy support for cookies. When you use cookies so that you are logged into youtube, the error 429 does not occur anymore.

[mahhov]

For what it's worth, @fent and @akanellis 's solution of using cookies worked, at least for now. I just hope my account doesn't get blocked or something.

I navigated to youtube.com in chrome, opened dev console, and copied the cookies from a network request.
Then do:

ytdl(<id>, {
            ...
            requestOptions: {
                headers: {
                    'Cookie': '<paste cookie as string>'
                }
            }
        })

[mahhov]

Apparently, even using oauth isn't immune to 429.
Using oauth in the request options (e.g. {filter:'audioonly', requestOptions: {headers: {Authorization: 'Bearer ...'}}}gives me ~15-25 videos, before getting another 429.
Refreshing the access token (but using the same refresh token and thereby not requiring user re-login) gives me another 15-25 videos.
This is just ~1 video per 3-5 minutes.
(I'm only 75% this comment is accurate :), I'll update it once I verify it again)

[mahhov]

Confirmed, my above comment is accurate. Refreshing the oath token does resolve the issue. It took 20 downloads to reach http 429. I guess something like this should work then

ytdl(...)
  .on('error', error => {
    if (error.message === 'Status code: 429') {
      // refreh token
      // try again
    }
  })

[eric-mathison]

I'm 429 blocked now too so I've been playing around with cookies. I've narrowed it down to these three:
'SID=xxx; HSID=xxx; SSID=xxxx;'

I've been able to make successful calls after setting these manually.
They are all set to expire in July 2022, so hopefully I can get some good use out of them until then.

[WaqasIbrahim]

@eric-mathison Were you able to make requests from a blocked IP?

[eric-mathison]

@eric-mathison Were you able to make requests from a blocked IP?

I couldn't confirm that my IP was blocked, but I kept getting error 429 from my node application running local on my laptop. I waited over 14 hours and it was still receiving this error. Passing those 3 cookies is working. I just finished downloading 52 videos in around 7 minutes time. They are small 75mb and I downloaded them slow, 1 at a time. So far so good.

[WaqasIbrahim]

Does not seem to work on a server that's already blocked.

Edit: Same request works fine in Postman.

[fent]

Does not seem to work on a server that's already blocked.

Edit: Same request works fine in Postman.

it's showing 10 headers in postman, maybe that's helping

[WaqasIbrahim]

@fent Those are standard headers, I generated the curl command using postman.

Update:
I disabled IPv6 on the server.
Without headers and cookie:

With headers and cookie

Update 2:
Did not last long, went back to 429.

[eric-mathison]

Does not seem to work on a server that's already blocked.

Edit: Same request works fine in Postman.

@WaqasIbrahim that doesn't look like an IP block. You are getting blocked by reCaptcha. See if you can curl the get_video_info url directly. YouTube doesn't block the watch url. Only the get info.

[WaqasIbrahim]

@eric-mathison get_video_info seems to be working but watch page URL is used first. get_video_info is a fallback URL.

[eric-mathison]

@eric-mathison get_video_info seems to be working but watch page URL is used first. get_video_info is a fallback URL.

Eh, you're probably right. I was just able to get it working from the get_video_info URL since the watch pages never broke for me.

[Abhay07]

My IP was blocked and i can confirm that. setting proxy or anything didn't work. Had to spin up a new server.

[angelhodar]

there's already a memory cache i place, but it's currently for 1 sec, and currently non configurable

So that would be like a throttling mechanism right? What if people could configure that caching time? I have seen the youtube-dl repo and they only use caching for saving some time when decrypting video info but not the already requested video data.

[orfox32]

does anyone know how to fix the error 429 coz i got it again after 1 week even though i have a cookie

[twarped]

actually, i ended up using youtube-dl. after messing around, I accidentally came up with this solution:
youtubedl.getInfo(videoUrl, function(err, info) {
res.header("Content-Disposition", "attachment; filename=\""+info.title+".mp4\""); //filename optional. either use content-disposition or res.write("") to make it download directly to response
res.write("")//without this (if you don't have content-disposition), it just streams the video within the browser
require("request").get(info.url).pipe(res)
});
i tested it about 50 times without coming about with 429
I couldn't get 429

here's my code/example: (https://tattered-lead-archaeopteryx.glitch.me/)

[redbrain]

Bottom line, for all you newcomers: 429 is YouTube trying to limit you. It is not a problem with the ytdl-core project.
Our friend Promise in the Discord guild pointed out the following solutions:

1. Make sure you're at the latest version of ytdl-core
2. Try using cookies (example in examples folder) This works quite well most of the time.
3. Try using proxies (example in examples folder) This is difficult to set up but mostly foolproof
4. Switch to IPv6 (if you aren't already) Block-rotating is coming soon, plus it gives you a new address upfront.
5. If all else fails, just wait between a few hours and a few days, and it will go away.

fent, I don't know how GitHub Issues works, but perhaps place these pointers somewhere important so we don't get more people asking the same question.

[billpan04]

Can I get a detailed explanation on how to use cookies within my code? because I did not understand a thing with the little to no explanation provided in the examples folder

[redbrain]

The file examples/cookies.js is an example of code that could be used to provide a cookie to a ytdl request, as well as instructions on how to obtain a cookie from your browser.

[billpan04]

I know... I asked for a detailed explanation because the explanation that the file had were so little that I couldnt understand anything

[redbrain]

To use the YouTube cookie, you get it from your browser and put it as a string in ytdl's requestOptions.headers.cookie.

Lines 6-12 of the file contain very thorough instructions on how to find that cookie. From there you just paste the cookie into the request options as I've described, wherever you use a ytdl requesting function, such as ytdl() or getInfo().

I've basically just restated what that file said, because that's literally all possible information you would need to implement it.
If you still don't understand something, you'll have to explain what exactly you don't understand.

[billpan04]

Thank you I managed to figure out how to use cookies. For anyone with the same problem take a look at #661 it might help understand how to obtain and add them to your code.

[twarped]

Do people still get 429? I just downloaded 100 videos as a test and I never got it... Maybe Youtube changed their systems

[mpirescarvalho]

Yes, I do... sometimes... I'm using heroku, when it starts to happen I just restart the dyno and it works fine again

[InfiniteMarcus]

I'm using Heroku too and 429 still appears, even with cookies. Restarting prevents it for some time, but unfortunately is not a solution.

[twarped]

hmm... i have a fork that im using, it doesn't have full audio and quality options yet, but you could try using it, https://github.com/twarped/node-ytdl-core/tree/patch-3
https://tattered-lead-archaeopteryx.glitch.me is my code example... if you want to see that...

[YeyintThway13]

• How can I prevent this? I'd like to have some sort of rate throttling so it doesn't incur into 429 errors. It renders ytdl completely unusable for a long time (no exact amount, last time it was more than an hour)

I'm going to be adding throttling so that it doesn't make requests too fast to prevent these errors

• No error is thrown by ytdl whatsoever. This is not good because there is no way to handle the error. The callback never gets invoked, and error handlers are never called.

will be fixed soon

Still got these errors

[EPXOL]

Hello I looked into this problem and find solution go to node_modules/lib/info.js in there find function named getBasicInfo you can see somethink like this in it:

  let info = await pipeline([id, options], validate, retryOptions, [
    getWatchHTMLPage,
    getWatchJSONPage,
    getVideoInfoPage,
  ]);

rewrite it to

  let info = await pipeline([id, options], validate, retryOptions, [
    getWatchHTMLPage,
    getVideoInfoPage,
  ]);

WatchHTMLPage is simular to WatchJSONPage it does not efect any of data everything will be downloaded from Watch Page

Why does this work ?
It works becouse JSONPage uses access token to make request and that is cousing too many requests error that meens that if we disable request to JSONPage access token will not be used and there will not be any more too many requests error.

This works with downloading of streams too becouse when you are downloading stream ytdl-core uses getInfo function to get video URLS.

[redbrain]

Can we confirm that watchJSONPage is completely obsoleted by watchHTMLPage, or is it needed for some things?

[EPXOL]

I can confirm that it's basicly the same becouse watchJSONPage is basicly just raw JSON from watchHTMLPage you can go to source of package and try it yourself. Basicly all of these 3
getWatchHTMLPage,
getWatchJSONPage,
getVideoInfoPage,
Are the same. It returns same data.

[EPXOL]

Btw fixed new bug of ytdl-core when youtube updated API #939 (comment) Includes this fix within it as well.

[securing-ab]

Nope,it isn’t fixed,I’m still getting the error.

[smironescu]

Seems that on our server the errors 429 had started but locally, with the same cookie all is working fine :S.

[Instinzts]

Thank you I managed to figure out how to use cookies. For anyone with the same problem take a look at #661 it might help understand how to obtain and add them to your code.

Mind, Giving Me An Example How To Do It The Example Provided inst enough for me to understand

[github-actions]

🎉 This issue has been resolved in version 4.11.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[YueMiyuki]

Please check this: https://www.npmjs.com/package/ytcr
:D