[YouTube] Revive age-gate bypass

[dirkf]

Boilerplate: mixed code, improvement

## Please follow the guide below

• You will be asked some questions, please read them carefully and answer honestly
• Put an x into all the boxes [ ] relevant to your pull request (like that [x])
• Use Preview tab to see how your pull request will actually look like

---

Before submitting a pull request make sure you have:

• [x] Searched the bugtracker for similar pull requests
• [x] Read adding new extractor tutorial
• [x] Read youtube-dl coding conventions and adjusted the code to meet them
• [x] Covered the code with tests (note that PRs without tests will be REJECTED)
• [x] Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

• [x] I am the original author of this code and I am willing to release it under Unlicense, except where
• [x] I am not the original author of this code but it is in public domain or released under Unlicense (https://github.com/yt-dlp/yt-dlp/pull/3233)

What is the purpose of your pull request?

• [ ] Bug fix
• [x] Improvement
• [ ] New extractor
• [ ] New feature

---

Description of your pull request and other information

This PR revives the ability to bypass some age restrictions on YT videos (specifically tiers 1-3 of https://github.com/yt-dlp/yt-dlp/pull/575#issuecomment-888837000), following yt-dlp PR #3233.

This function is disabled if the --age-limit parameter is set to a value less than 18.

Added with this PR is utility function update_url(url, **kwargs) which allows modifying (eg, ..., netloc='www.youtube.com')) or removing (eg, ..., query=None)) URL components from a URL string or parsed URL tuple to produce a new URL string.

Resolves #1495 Resolves #5617 Resolves #26652 Resolves #26665 Resolves #29578 Resolves #29086 Resolves #29659 Resolves #29729 Resolves #29811 Resolves #30019 Resolves #30255 Resolves #30953 Resolves #31034

[nicolaasjan]

Could you tell me what is the reason why this PR still isn't merged into master ?

[dirkf]

Could you tell me what is the reason why this PR still isn't merged into master ?

Just things to do. Let me know if you've had success with it.

[nicolaasjan]

Could you tell me what is the reason why this PR still isn't merged into master ?

Just things to do. Let me know if you've had success with it.

Your commits added to the most recent youtube-dl master worked just fine, without the need to pass cookies, with a random number of video's from the list of issues you gave as resolved by it. 🙂 (I didn't apply your recent suggested change yet)

[nicolaasjan]

@dirkf Maybe I'm just dumb (I'm not a programmer...), but for some reason this patch doesn't apply any more:

git apply 31043.patch
error: patch failed: youtube_dl/extractor/youtube.py:1545
error: youtube_dl/extractor/youtube.py: patch does not apply

Would you be so kind to enlighten me?

[dirkf]

Patch may not work, but merging should, or the box above wouldn't say

This branch has no conflicts with the base branch Merging can be performed automatically.

Anyhow, I've updated from the latest master, so you should be able to get a good patch now.

[nicolaasjan]

Patch file still didn't apply.

This worked:

git pull origin pull/31043/head:df-yt-agegate-patch
Merge made by the 'recursive' strategy.
 youtube_dl/extractor/youtube.py | 234 ++++++++++++++++++++++++++++++++--------
 youtube_dl/utils.py             |  12 +++
 2 files changed, 200 insertions(+), 46 deletions(-)

Hope this "strategy" doesn't mess things up. 😀️

[nicolaasjan]

Is this something to worry about (I use this pull request in my custom build):

Some checks were not successful 23 cancelled, 3 failing, and 1 successful checks

[dirkf]

It was, but only temporarily. I wanted to migrate some code improvements needed for the PR to library modules before merging and that set off a chain of dependent/consequent improvements (previous maintainers might frown).

Please pass on any feedback from the current commit.

[nicolaasjan]

Please pass on any feedback from the current commit.

No issues I think. 🙂

youtube-dl -v p7FCgw_GlWc
[debug] System config: []
[debug] User config: ['-i', '-o', '/dev/shm/test-ytd/%(title)s.%(ext)s', '-f', 'bestvideo[height<=1080][ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best', '--no-mtime', '--embed-thumbnail', '--force-ipv4']
[debug] Custom config: []
[debug] Command-line args: ['-v', 'p7FCgw_GlWc']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2023.02.07
[debug] Python version 3.8.10 (CPython) - Linux-5.4.0-137-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg N-109587-gfc263f073e-Nico-20230113, ffprobe N-109587-gfc263f073e-Nico-20230113, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] p7FCgw_GlWc: Downloading webpage
[youtube] Confirming age
[youtube] p7FCgw_GlWc: Downloading API JSON
[debug] [youtube] Decrypted nsig TyseLXL-R_G1uorxBrO => CTXoSiZPjD5V0g
[debug] [youtube] Decrypted nsig D0aeHVrxx_314ybOSYF => vwcjwcqyF47CrQ
[youtube] p7FCgw_GlWc: Downloading thumbnail ...
[youtube] p7FCgw_GlWc: Writing thumbnail to: /dev/shm/test-ytd/Kanye West - Famous.jpg
[debug] Invoking downloader on 'https://rr1---sn-5hneknee.googlevideo.com/videoplayback?expire=1675768762&ei=Wt_hY9W5A9mE8gOE2ILIBw&ip=<REDACTED>&id=o-AA0wBznLVxNBc-GM0VIwzDmCBU_Iw3s3HkDwV6MUf5HZ&itag=137&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%2C394%2C395%2C396%2C397%2C398%2C399&source=youtube&requiressl=yes&mh=a3&mm=31%2C26&mn=sn-5hneknee%2Csn-4g5edndr&ms=au%2Conr&mv=m&mvi=1&pl=17&gcr=nl&initcwndbps=1937500&vprv=1&mime=video%2Fmp4&ns=TTeFNo14CK--wVcF-sbD0t0L&gir=yes&clen=226969479&dur=636.902&lmt=1637456618561210&mt=1675746549&fvip=3&keepalive=yes&fexp=24007246&beids=24239132&c=TVHTML5_SIMPLY_EMBEDDED_PLAYER&txp=5535434&n=CTXoSiZPjD5V0g&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cgcr%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAPdNAhHz5JE23zIAhMCTZRyGyAuqsfXpCzJ7BWDEdHAjAiEA1-GlamAKXebxPzXYxzBcAuF7y7yFbSZt5U9FQJD46o4%3D&sig=AOq0QJ8wRQIhAP1zb-E8O9ExSaDdaf6A2ngHxDSlvQPbEGvj8o1YiKPyAiBIrD2D4mA_jmS7Utb5dEGV0y31SU0BKgeYGDDgwaFuGA%3D%3D'
[download] Destination: /dev/shm/test-ytd/Kanye West - Famous.f137.mp4
[download] 100% of 216.45MiB in 00:19
[debug] Invoking downloader on 'https://rr1---sn-5hneknee.googlevideo.com/videoplayback?expire=1675768762&ei=Wt_hY9W5A9mE8gOE2ILIBw&ip=<REDACTED>&id=o-AA0wBznLVxNBc-GM0VIwzDmCBU_Iw3s3HkDwV6MUf5HZ&itag=140&source=youtube&requiressl=yes&mh=a3&mm=31%2C26&mn=sn-5hneknee%2Csn-4g5edndr&ms=au%2Conr&mv=m&mvi=1&pl=17&gcr=nl&initcwndbps=1937500&vprv=1&xtags=drc%3D1&mime=audio%2Fmp4&ns=TTeFNo14CK--wVcF-sbD0t0L&gir=yes&clen=10310099&dur=636.992&lmt=1655987080855281&mt=1675746549&fvip=3&keepalive=yes&fexp=24007246&beids=24239132&c=TVHTML5_SIMPLY_EMBEDDED_PLAYER&txp=5532434&n=CTXoSiZPjD5V0g&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cgcr%2Cvprv%2Cxtags%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIgUgJdHqr3LUgJHogzhP21q1TYsXAfUybVECjPxeCI8xQCIQDctfpjvyGjSNruZZ-9WmvaVb_WuplevuxjdOVQaNU3HQ%3D%3D&sig=AOq0QJ8wRQIgf2VCdnCA4_UuGclxKMLB2tC4fY2mt45fVb3nMWd0hPoCIQD3zl0gQJPYIWJvyBL_5rD8WRoB7v3c3CrRdBlHwYDgZg%3D%3D'
[download] Destination: /dev/shm/test-ytd/Kanye West - Famous.f140-1.m4a
[download] 100% of 9.83MiB in 00:00
[ffmpeg] Merging formats into "/dev/shm/test-ytd/Kanye West - Famous.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/dev/shm/test-ytd/Kanye West - Famous.f137.mp4' -i 'file:/dev/shm/test-ytd/Kanye West - Famous.f140-1.m4a' -c copy -map 0:v:0 -map 1:a:0 'file:/dev/shm/test-ytd/Kanye West - Famous.temp.mp4'
Deleting original file /dev/shm/test-ytd/Kanye West - Famous.f137.mp4 (pass -k to keep)
Deleting original file /dev/shm/test-ytd/Kanye West - Famous.f140-1.m4a (pass -k to keep)
[ffmpeg] Adding thumbnail to "/dev/shm/test-ytd/Kanye West - Famous.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:/dev/shm/test-ytd/Kanye West - Famous.mp4' -i 'file:/dev/shm/test-ytd/Kanye West - Famous.jpg' -c copy -map 0 -map 1 -disposition:v:1 attached_pic 'file:/dev/shm/test-ytd/Kanye West - Famous.temp.mp4'

Note, that I added pull requests #29581 and #29593 as well (to use FFmpeg instead of AtomicParsley for thumbnail embedding).

[yermulnik]

Is it that to get a youtube-dl with this patch the only path forward is to git clone .. && make install or there is a place we can get a latest release, which is not mentioned in README? Thanks.

[dirkf]

When you update yt-dl to fix #31530 (not by switching to yt-dlp, obvs, though yt-dlp has fuller functionality for this anyway), this commit will be included too.

[yermulnik]

@dirkf

When you update yt-dl

What's the recommended way of doing this given there have been no new releases since late 2021? Is it git clone ... && make install or there's another approach? Thanks.

[dirkf]

Read the issue!