feat: add IPv6 block rotating #713

Roki100 · 2020-09-28T15:54:54Z

This feature was coded in a collab with @Million900o
Hello! I would like to present the new feature(s):

Rotating trough IPv6 block to stop YouTube IP Bans (/58 block or larger recommended, however /64 should work aswell)
Ability to disable chunking, as it is not needed while streaming in 1x speed (in for example Discord bots)

How does the method implemented work: in short, it uses different IPv6 address each time you download/request something via ytdl, for larger blocks(that include more that one /64) it also uses random /64 to make IP Bans impossible to get.

Feel free to propose changes, as this code is for sure not the cleanest

Note: The IPv6 rotating will work only if the block provided is owned by you and configured in the OS!
For setting up guide i recommend reading:
https://contabo.com/blog/adding-ipv6-connectivity-to-your-server
https://contabo.com/blog/configuring-additional-ip-addresses/
https://support.us.ovhcloud.com/hc/en-us/articles/360002296020-How-to-Configure-an-IPv6-Address-in-Ubuntu
Or googling for more tutorials.

Also note: This functionality has been tested with /64, /58 and /48 blocks, and it works, however its a first version and i will keep improving that.
Adding new ytdl dependency was needed because its kinda complex/huge math to get a random ipv6 subnet.

closes #635

Inspired by Lavalink's rotator.

Roki100 · 2020-09-28T15:57:52Z

I would like to also give our test results about IP Bans (tested on a 6.5k+ servers discord music bot):

IPv4: ~2 days
IPv6 /64: none
IPv6 /58: none
IPv6 /48: none

Fixing the format of files so eslint does not throw any errors.

TimeForANinja · 2020-09-28T16:30:51Z

I guess you already spottet that the CI asks you to add tests for the new code lines 😉

alexanderpaolini · 2020-09-28T16:39:29Z

How do we make tests?

Roki100 · 2020-09-28T17:30:20Z

Done

test/info-test.js

fent · 2020-10-03T07:40:55Z

hi, thanks for the PR Roki and Million900o! it's pretty impressive that there are no bans with this on your bot during your testing. how long have you ran it for using the PR?

Rotating trough IPv6 block to stop YouTube IP Bans (/58 block or larger recommended, however /64 should work aswell)

Ability to disable chunking, as it is not needed while streaming in 1x speed (in for example Discord bots)

I'm going to start looking at this within the next day or so, but I will say that generally a PR is one change. keeping PRs small and contained to their scope not only makes the review process simpler, but it makes history cleaner, release changes easier to list, and if we ever have to git bisect or git revert we can do that per PR.

please separate the above two changes into 2 PRs. I'd keep this one, remove any changes about disabling chunking since those changes should be smaller than IPv6 rotating, and create another one with only the chunking changes. but also, chunking can be disabled with dlChunkSize: Infinity, or is that not the same?

Roki100 · 2020-10-03T15:28:44Z

hi, thanks for the PR Roki and Million900o! it's pretty impressive that there are no bans with this on your bot during your testing. how long have you ran it for using the PR?

Rotating trough IPv6 block to stop YouTube IP Bans (/58 block or larger recommended, however /64 should work aswell)

Ability to disable chunking, as it is not needed while streaming in 1x speed (in for example Discord bots)

I'm going to start looking at this within the next day or so, but I will say that generally a PR is one change. keeping PRs small and contained to their scope not only makes the review process simpler, but it makes history cleaner, release changes easier to list, and if we ever have to git bisect or git revert we can do that per PR.

please separate the above two changes into 2 PRs. I'd keep this one, remove any changes about disabling chunking since those changes should be smaller than IPv6 rotating, and create another one with only the chunking changes. but also, chunking can be disabled with dlChunkSize: Infinity, or is that not the same?

The bot currently runs on a /64 block for over a week now and there are no bans so far

Also, what do i do to remove chunking changes from this pr? like just remove them and make a commit or there is a command for that? I am not sure about the dlChunkSize: Infinity but chunking: false seems to be easier to understand and it completely "disables" the chunking part in the code

fent · 2020-10-03T18:02:28Z

Also, what do i do to remove chunking changes from this pr? like just remove them and make a commit or there is a command for that?

remove the changes for it and make a commit

I am not sure about the dlChunkSize: Infinity but chunking: false seems to be easier to understand and it completely "disables" the chunking part in the code

how about dlChunkSize: 0?

Roki100 · 2020-10-03T19:57:38Z

remove the changes for it and make a commit

will do in a sec

how about dlChunkSize: 0?

Hmm wouldnt it make ytdl go crazy?

      let start = (options.range && options.range.start) || 0;
      let end = start + dlChunkSize;

so like start and end will be the same

fent · 2020-10-03T20:08:18Z

Hmm wouldnt it make ytdl go crazy?
      let start = (options.range && options.range.start) || 0;
      let end = start + dlChunkSize;
so like start and end will be the same

oh i meant have it be turned off if it detects that dlChunkSize === 0

Roki100 · 2020-10-03T20:14:25Z

oh i meant have it be turned off if it detects that dlChunkSize === 0

Hmmm, i might change it to that then i guess(with addition to readme), but first of all i have to remove chunking stuff from this pr and open second one

Roki100 · 2020-10-03T20:22:00Z

Okay, i think i've removed every change for chunking

lib/util.js

test/download-test.js

test/info-test.js

Co-authored-by: fent <[email protected]>

TimeForANinja

boy typing with a broken hand suxx
next time i'll send a voice memo

TimeForANinja · 2021-03-21T20:51:39Z

README.md

@@ -149,11 +151,16 @@ ytdl cannot download videos that fall into the following
 * YouTube Premium content (if you have access, requires [cookies](example/cookies.js))

 Generated download links are valid for 6 hours, and may only be downloadable from the same IP address.
+Using IPv6 should not cause same IP address limitation.


did you check with

different ip from same /64 blocks or

even different /64 blocks?

i did check with multiple ipv4's and ipv6 in same /64

package.json

TimeForANinja · 2021-03-21T20:56:50Z

lib/utils.js

+ */
+exports.getRandomIPv6 = block => {
+  let subnet = parseInt(block.split('/')[1]);
+  block = block.split('/')[0];


i'm personally not a fan of reassigning to passed variables
also calling the expensive split function twice....
how about sth like

const [net, rawSubnet] = block.split('/'); const subnet = parseInt(rawSubnet)

?

TimeForANinja · 2021-03-21T21:01:42Z

lib/utils.js

+  block = block.split('/')[0];
+  if (!subnet || subnet > 128 || subnet < 24) throw Error('Invalid IPv6 subnet');
+  // Other errors will be thrown by ip6 itself
+  return ip6.randomSubnet(block, subnet, 128, 1, true)[0];


i see we're not first rolling the /64 net
like it

yea it was a requested change by you so i changed 👀

typings/index.d.ts

TimeForANinja · 2021-03-21T21:14:53Z

test/full-info-test.js

+  describe('With IPv6 Block', () => {
+    it('Sends request with IPv6 address', async() => {
+      const id = '_HSylqgVYQI';
+      const scope = nock(id, 'regular');
+      let info = await ytdl.getInfo(id, { IPv6Block: '2001:2::/48' });
+      nock.url(info.formats[0].url).reply(function checkAddr() {
+        // "this" is assigned by the function checkAddr
+        // eslint-disable-next-line no-invalid-this
+        assert.ok(net.isIPv6(this.req.options.localAddress));
+        scope.done();
+      });
+    });
+  });
+
+  describe('With invalid IPv6 Block', () => {
+    it('Should give an error', async() => {
+      const id = '_HSylqgVYQI';
+      await assert.rejects(ytdl.getInfo(id, { IPv6Block: '2001:2::/200' }), /Invalid IPv6 subnet/);
+    });
+  });


we could test the getRandomIPv6 (but not here) function instead of doing the same pass an ipv6 test (like above) twice

so that we do
a) test the whole usage via ytdl-core once
b) do white box tests only on the getRandomIPv6 function for error like
b.a) no subnet (2001:2::) or
b.b) subnet only (/64) or
b.c) passing errors from ip6.randomSubnet

TimeForANinja · 2021-05-27T20:18:41Z

if you wanna get rid of that ipv6 library 😉

const randomIP = (ip) => {
    const [addr, mask] = ip.split('/');
    // normalize inputs
    let base10Mask = parseInt(mask);
    const base10addr = normalizeIP(addr);
    // get random addr to pad with
    // using Math.random since we're not requiring high level of randomness
    const randomAddr = new Array(8).fill(1).map(x => Math.floor(Math.random() * 0xffff));
    // merge base10addr with randomAddr
    const mergedAddr = randomAddr.map((randomItem, idx) => {
        // calculate the amount of static bits
        const staticBits = Math.min(base10Mask, 16);
        // adjust the bitmask with the staticBits
        base10Mask -= staticBits;
        // calculate the bitmask
        // lsb makes the calculation way more complicated
        const mask = 0xffff - ((2**(16 - staticBits)) - 1);
        // combine base10addr and random
        return (base10addr[idx] & mask) + (randomItem & (mask ^ 0xffff));
    });
    // return new addr
    return mergedAddr.map(x => x.toString('16')).join(':');
}

const normalizeIP = (ip) => {
    // split by fill position
    const parts = ip.split('::').map(x => x.split(':'));
    // normalize start and end
    const partStart = parts[0] || [];
    const partEnd = parts[1] || [];
    // placeholder for full ip
    const fullIP = new Array(8).fill(0);
    // fill in start and end parts
    for (let i = 0; i < Math.min(partStart.length, 7); i++) {
        fullIP[i] = parseInt(partStart[i], 16) || 0;
    }
    for (let i = 0; i < Math.min(partEnd.length, 7) ; i++) {
        fullIP[7 - i] = parseInt(partEnd[i], 16) || 0;
    }
    return fullIP;
}

muhitrhn · 2021-06-27T00:35:21Z

Can anyone help me to setup IPv6 on AWS console and setting it up with? (I already have all the codes in place in the latest version.)

I really can’t get it working.

TimeForANinja · 2021-07-30T04:47:37Z

@Roki100 can you have another look, pls

Roki100 · 2021-07-30T06:58:16Z

@Roki100 can you have another look, pls

oh, yea sure, i have to come back from holiday tho as i dont have pc atm (3 days)

TimeForANinja · 2021-07-30T08:01:56Z

sure - just tell me when you had a look 😉

lib/utils.js

Co-authored-by: Voltrex <[email protected]>

stale · 2021-12-09T00:24:02Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Roki100 · 2021-12-13T15:22:31Z

@TimeForANinja aww, my bad on not responding, yeah everything seems to be fine 👍
Edit: Sorry again for being so late with the response here 😞

github-actions · 2022-03-05T14:57:42Z

🎉 This PR is included in version 4.11.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

IPv6 Rotating ^-^

67b56f4

Roki100 and others added 2 commits September 28, 2020 18:02

linter got mad at missing semicolon nice

4ff3b8c

Fixing format

1f0d264

Fixing the format of files so eslint does not throw any errors.

adding colon

88406fe

alexanderpaolini and others added 3 commits September 28, 2020 13:30

added test for IPv6 Block

a8399fd

Forgot to lint

3c1fee3

Info test and download test improvement

3436d37

TimeForANinja reviewed Sep 28, 2020

View reviewed changes

test/info-test.js Outdated Show resolved Hide resolved

use net instead if .includes

a2cb9d8

remove chunking-related stuff

d24ac56

fent requested changes Oct 4, 2020

View reviewed changes

lib/util.js Outdated Show resolved Hide resolved

lib/util.js Outdated Show resolved Hide resolved

lib/util.js Outdated Show resolved Hide resolved

test/download-test.js Outdated Show resolved Hide resolved

test/info-test.js Outdated Show resolved Hide resolved

Roki100 and others added 8 commits October 4, 2020 02:34

Update lib/util.js

627dfc5

Co-authored-by: fent <[email protected]>

improve tests, fix the check in util

916df54

add test for invalid subnet

ff44eb9

place done's on separate lines

1eef049

fix typo in readme

474cf88

fixing README.md

04e5cd9

fixed mistake

ef00f8b

fixed other mistake

7ad0ca7

alexanderpaolini added 2 commits March 19, 2021 15:01

Adding requested test

49efc57

Merge branch 'master' of https://github.com/Roki100/node-ytdl-core

01a9218

TimeForANinja reviewed Mar 21, 2021

View reviewed changes

Roki100 added 3 commits March 25, 2021 10:00

remove unrelated dependency

1565ad1

Merge branch 'master' of https://github.com/Roki100/node-ytdl-core

fa65f6e

remove unused option

fd0fdbe

TimeForANinja requested a review from fent April 10, 2021 21:52

TimeForANinja and others added 6 commits July 30, 2021 06:37

update readme

2e4e100

lint ipv6 example

bba847f

remove ip6 dependencie

f8c176a

improve test coverage

b51c799

(es)lint

5eb7d0f

Merge branch 'master' into master

b46d23a

VoltrexKeyva reviewed Sep 11, 2021

View reviewed changes

lib/utils.js Outdated Show resolved Hide resolved

Update lib/utils.js

baef233

Co-authored-by: Voltrex <[email protected]>

stale bot added the stale Issues that were closed for inactivity label Dec 9, 2021

stale bot removed the stale Issues that were closed for inactivity label Dec 13, 2021

TimeForANinja merged commit 06f5611 into fent:master Mar 5, 2022

github-actions bot added the released label Mar 5, 2022

dskaiser82 mentioned this pull request Aug 21, 2022

Fix: Invalid regular expression: missing / #1110 #1111

Closed

adityapanchal10 mentioned this pull request May 29, 2024

[Snyk] Upgrade ytdl-core from 4.8.3 to 4.11.5 adityapanchal10/vinylizer#38

Open

adityapanchal10 mentioned this pull request Sep 13, 2024

[Snyk] Upgrade: , ffmpeg-static, yt-search, ytdl-core, ytpl adityapanchal10/vinylizer#45

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add IPv6 block rotating #713

feat: add IPv6 block rotating #713

Roki100 commented Sep 28, 2020 •

edited by fent

Loading

Roki100 commented Sep 28, 2020 •

edited

Loading

TimeForANinja commented Sep 28, 2020

alexanderpaolini commented Sep 28, 2020

Roki100 commented Sep 28, 2020

fent commented Oct 3, 2020

Roki100 commented Oct 3, 2020

fent commented Oct 3, 2020

Roki100 commented Oct 3, 2020

fent commented Oct 3, 2020

Roki100 commented Oct 3, 2020

Roki100 commented Oct 3, 2020

TimeForANinja left a comment

TimeForANinja Mar 21, 2021

Roki100 Mar 25, 2021

TimeForANinja Mar 21, 2021

TimeForANinja Mar 21, 2021

Roki100 Mar 25, 2021

TimeForANinja Mar 21, 2021

TimeForANinja commented May 27, 2021

muhitrhn commented Jun 27, 2021 •

edited

Loading

TimeForANinja commented Jul 30, 2021

Roki100 commented Jul 30, 2021

TimeForANinja commented Jul 30, 2021

stale bot commented Dec 9, 2021

Roki100 commented Dec 13, 2021 •

edited

Loading

github-actions bot commented Mar 5, 2022

feat: add IPv6 block rotating #713

feat: add IPv6 block rotating #713

Conversation

Roki100 commented Sep 28, 2020 • edited by fent Loading

Inspired by Lavalink's rotator.

Roki100 commented Sep 28, 2020 • edited Loading

TimeForANinja commented Sep 28, 2020

alexanderpaolini commented Sep 28, 2020

Roki100 commented Sep 28, 2020

fent commented Oct 3, 2020

Roki100 commented Oct 3, 2020

fent commented Oct 3, 2020

Roki100 commented Oct 3, 2020

fent commented Oct 3, 2020

Roki100 commented Oct 3, 2020

Roki100 commented Oct 3, 2020

TimeForANinja left a comment

Choose a reason for hiding this comment

TimeForANinja Mar 21, 2021

Choose a reason for hiding this comment

Roki100 Mar 25, 2021

Choose a reason for hiding this comment

TimeForANinja Mar 21, 2021

Choose a reason for hiding this comment

TimeForANinja Mar 21, 2021

Choose a reason for hiding this comment

Roki100 Mar 25, 2021

Choose a reason for hiding this comment

TimeForANinja Mar 21, 2021

Choose a reason for hiding this comment

TimeForANinja commented May 27, 2021

muhitrhn commented Jun 27, 2021 • edited Loading

TimeForANinja commented Jul 30, 2021

Roki100 commented Jul 30, 2021

TimeForANinja commented Jul 30, 2021

stale bot commented Dec 9, 2021

Roki100 commented Dec 13, 2021 • edited Loading

github-actions bot commented Mar 5, 2022

Roki100 commented Sep 28, 2020 •

edited by fent

Loading

Roki100 commented Sep 28, 2020 •

edited

Loading

muhitrhn commented Jun 27, 2021 •

edited

Loading

Roki100 commented Dec 13, 2021 •

edited

Loading