Bandwidth throttling mitigation

[Serkora]

Before submitting a pull request make sure you have:

• [x] At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
• [x] Searched the bugtracker for similar pull requests

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

• [x] I am the original author of this code and I am willing to release it under Unlicense
• [ ] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

• [ ] Bug fix
• [x] Improvement
• [ ] New extractor
• [x] New feature

---

Description of your pull request and other information

#6923 Pretty much self explanatory. Some websites only let you download a little bit at maximum speed and then slow it down to some value (may be media bitrate, may be just an arbitrary limit they decided upon) If this website also supports Range header, a new request can be made to get full speed again. Repeat until ready.

In a nutshell, that's all this code does: monitors the overall (to get the peak) and separate block transfer rates and makes a new request when the block rate falls below a certain threshold. And a new option is added, of course. A few unnecessary reconnections here and there, but mostly stable chunks are downloaded before each new request is made, so I'm rather pleased with it.

Important notes:

1. When disabled, downloader behaves exactly as before (other than a few excessive calcuations, perhaps), so it should be safe to replace the original downloader with it (instead of a making a separate one).
2. I can't (easily) simulate various network conditions, so more testing is required to see if the strategies used are actually good and robust enough.
3. Tested mainly in clear network conditions and on two websites only, with high and not so high peak rates (5-6MiB/s and ~1.2MiB/s) and similar throttling rates (50-100KiB/s), so all of the constant values and dynamic formulae, while thought out, are approximate and obtained partially empirically to maximaize the overall mean download rate on my computer. ymmv, as always. Any suggestions are welcomed, testing of various different websites is greatly appreciated,
4. Mean download rate of around 50-80% of the reported peak can be achieved,
5. It behaves quite well in absence of throttling, only a few spurious reconnections might occur due to temporary network issues. Not with a bad internet, however.

Boring explanations of the strategies, can be easily seen in code but just so that it is documented: First of all, no way to tell if throttling started or your network became bad (or downloading lots of porn in parallel, for instance), so if you start a second network-intesive task, well, things will break. The main thing with throttling is the block size. Too small — slow overall download and/or unreliable instantaneous rate measurements. Too large — you'll become old before throttling is detected. 512KiB seemed like a reasonable enough compromise to set as a starting point.

1. First second of the download is ignored, nothing can happen there,
2. Peak rate is only set if block size is 64KiB or more, because if it's less, block rate measurements will be very inaccurate (orders of magnitute) and this also means your connections is not probably not good enough to suffer from throttling,
3. When peak rate is set and current block rate falls below 70% of that peak, timer starts,
4. After each consequent block with rate below 70%, a check is made to see if 3 seconds have passed, at which point it is assessed if the download rate starting from time recorded in point 2 is still below 70%. This helps deal with false positives when those blocks are a good amount of time apart from each other,
5. If throttling, the block size is set to the power of two nearest to the download rate during throttling, to download as little as possible during each subsequent throttle (i.e. detect it fast),
6. Threshold is set to the first quarter between throttling and peak rates (e.g. 400K peak and 50K throttling wil result in a 137.5K, or ~0.35 threshold) to cactch it early (e.g. if it was enabled in the middle of the block). However, unless you have shitty internet, throttling rate will be almost negligible, leading to the threshold of ~0.25-7
7. When this threshold is acquired, after each block a check is made to see if the current block rate fell below this threshold, and a new request is issued if so, replacing the data variable. Voila, grab some pizza and enjoy the thing you've just downloaded!

[Serkora]

Some verbose logs of downloads.

Youtube.com (no throttle):

$ youtube-dl --avoid-throttling -f 22 -v "https://www.youtube.com/watch?v=NYVjIjBMx6o"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--avoid-throttling', '-f', '22', '-v', 'https://www.youtube.com/watch?v=NYVjIjBMx6o']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.12.09
[debug] Python version 3.5.2 - Darwin-15.4.0-x86_64-i386-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[youtube] NYVjIjBMx6o: Downloading webpage
[youtube] NYVjIjBMx6o: Downloading video info webpage
[youtube] NYVjIjBMx6o: Extracting video information
[youtube] NYVjIjBMx6o: Downloading MPD manifest
[debug] Invoking downloader on 'https://r1---sn-gvnuxaxjvh-c35z.googlevideo.com/videoplayback?id=o-APYSVSXXi8KzfLVPFpoSMrZb6l-IA65DkTzGDxd3MUVN&ei=VDlWWJTuKtHLYsbcotgF&ms=au&mt=1482045426&mv=m&pl=36&ratebypass=yes&ip=2a02%3A2168%3A1840%3Af00%3A345a%3A70f9%3Adea9%3Ac916&source=youtube&mm=31&mn=sn-gvnuxaxjvh-c35z&sparams=dur%2Cei%2Cgcr%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cupn%2Cexpire&expire=1482067380&lmt=1478674677018311&key=yt6&initcwndbps=2705000&signature=DA81FF8898D65F071D91942FFF18C4D1C4800C00.D3BEC1269D8B9B44392B3557A282AE557911B220&ipbits=0&upn=hPqNnbnTLzI&itag=22&gcr=ru&mime=video%2Fmp4&dur=4798.403&requiressl=yes'
[download] Destination: New Macbook Pro can't walk & chew gum at same time(watch wifi + USB = fail)  PART 1-NYVjIjBMx6o.mp4
[download] 100% of 693.10MiB in 02:01

Vk.com:

$ youtube-dl -f url480 -u PRIVATE -o healer.mp4 -v --avoid-throttling "https://vk.com/video-41931658_171186378"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-f', 'url480', '-u', 'PRIVATE', '-o', 'healer.mp4', '-v', '--avoid-throttling', 'https://vk.com/video-41931658_171186378']
Type account password and press [Return]: 
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.12.09
[debug] Python version 3.5.2 - Darwin-15.4.0-x86_64-i386-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[vk] Downloading login page
[vk] Logging in as PRIVATE
[vk] -41931658_171186378: Downloading webpage
[debug] Invoking downloader on 'https://cs543604.vk.me/u3184474/videos/fed523eff7.480.mp4?extra=LMTmdjep5zkOloEtxfqFY5j2dGoE7cwQ_9BJB3GXBpOod0Q8-nascU18WK7mFymeqj6E5CzWzC2aX9VtTkrpOAJ72DfiN50C8X6IXly2UxYoqcUKhHsMdFB9DfhFoo7K9PRBUQLk6KgzHmgQnZUKyYm3HkhSvozEA3SJLzBBpgOiEw'
[download] Destination: healer.mp4
[download]   6.8% of 440.98MiB at  5.50MiB/s ETA 01:14
[throttling] Throttling started or is continuing, block rate = 803.15KiB/s, peak rate = 5628.14KiB/s
[download]   6.9% of 440.98MiB at  5.00MiB/s ETA 01:22
[throttling] Throttling started or is continuing, block rate = 234.91KiB/s, peak rate = 5628.14KiB/s
[download]   7.1% of 440.98MiB at  3.76MiB/s ETA 01:48
[throttling] Throttling started or is continuing, block rate = 223.10KiB/s, peak rate = 5628.14KiB/s
[throttling] Throttling detected! peak rate = 5628.14KiB/s, current rate = 348.02KiB/s, setting threshold to 0.30 and block size limit to 256KiB
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 223.10KiB/s, downloaded 32511KiB before throttling)
[download]  14.0% of 440.98MiB at  3.75MiB/s ETA 01:41
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 633.30KiB/s, downloaded 30976KiB before throttling)
[download]  20.9% of 440.98MiB at  4.13MiB/s ETA 01:24
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 652.57KiB/s, downloaded 30976KiB before throttling)
[download]  27.7% of 440.98MiB at  4.37MiB/s ETA 01:13
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 322.69KiB/s, downloaded 30976KiB before throttling)
[download]  34.6% of 440.98MiB at  4.40MiB/s ETA 01:05
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 788.02KiB/s, downloaded 30976KiB before throttling)
[download]  41.4% of 440.98MiB at  4.54MiB/s ETA 00:56
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 325.08KiB/s, downloaded 30976KiB before throttling)
[download]  48.3% of 440.98MiB at  4.44MiB/s ETA 00:51
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 370.88KiB/s, downloaded 30976KiB before throttling)
[download]  55.2% of 440.98MiB at  4.47MiB/s ETA 00:44
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 797.73KiB/s, downloaded 30976KiB before throttling)
[download]  62.0% of 440.98MiB at  4.55MiB/s ETA 00:36
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 283.98KiB/s, downloaded 30976KiB before throttling)
[download]  68.9% of 440.98MiB at  4.58MiB/s ETA 00:29
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 440.35KiB/s, downloaded 30976KiB before throttling)
[download]  75.7% of 440.98MiB at  4.62MiB/s ETA 00:23
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 576.62KiB/s, downloaded 30976KiB before throttling)
[download]  82.7% of 440.98MiB at  4.67MiB/s ETA 00:16
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 110.74KiB/s, downloaded 31488KiB before throttling)
[download]  89.6% of 440.98MiB at  4.59MiB/s ETA 00:09
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 319.12KiB/s, downloaded 31104KiB before throttling)
[download]  96.5% of 440.98MiB at  4.57MiB/s ETA 00:03
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 663.63KiB/s, downloaded 30976KiB before throttling)
[download] 100% of 440.98MiB in 01:36

pornhub.com:

$ youtube-dl --avoid-throttling -f 480P-600K -o a.mp4 -v "ADULT_CONTENT"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--avoid-throttling', '-f', '480P-600K', '-o', 'a.mp4', '-v', 'ADULT_CONTENT']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.12.09
[debug] Python version 3.5.2 - Darwin-15.4.0-x86_64-i386-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[PornHub] ph574abb7e735a1: Downloading webpage
WARNING: unable to extract uploader; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract view count; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract comment count; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[debug] Invoking downloader on 'ADULT_CONTENT'
[download] Destination: a.mp4
[download]   3.5% of 71.05MiB at  1.91MiB/s ETA 00:35
[throttling] Throttling started or is continuing, block rate = 92.54KiB/s, peak rate = 1952.35KiB/s
[throttling] Throttling detected! peak rate = 1952.35KiB/s, current rate = 46.26KiB/s, setting threshold to 0.27 and block size limit to 32KiB
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 92.54KiB/s, downloaded 3071KiB before throttling)
[download]   4.6% of 71.05MiB at 462.62KiB/s ETA 02:30
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 242.84KiB/s, downloaded 320KiB before throttling)
[download]   8.5% of 71.05MiB at 658.93KiB/s ETA 01:41
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 60.86KiB/s, downloaded 2848KiB before throttling)
[download]  12.4% of 71.05MiB at 740.34KiB/s ETA 01:26
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 126.56KiB/s, downloaded 2848KiB before throttling)
[download]  16.3% of 71.05MiB at 818.05KiB/s ETA 01:14
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 103.47KiB/s, downloaded 2816KiB before throttling)
[download]  20.2% of 71.05MiB at 858.01KiB/s ETA 01:07
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 152.97KiB/s, downloaded 2848KiB before throttling)
[download]  24.1% of 71.05MiB at 907.23KiB/s ETA 01:00
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 55.12KiB/s, downloaded 2848KiB before throttling)
[download]  28.1% of 71.05MiB at 928.58KiB/s ETA 00:56
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 59.97KiB/s, downloaded 2848KiB before throttling)
[download]  32.0% of 71.05MiB at 948.01KiB/s ETA 00:52
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 58.40KiB/s, downloaded 2848KiB before throttling)
[download]  35.9% of 71.05MiB at 955.26KiB/s ETA 00:48
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 104.96KiB/s, downloaded 2848KiB before throttling)
[download]  39.8% of 71.05MiB at 969.22KiB/s ETA 00:45
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 112.48KiB/s, downloaded 2848KiB before throttling)
[download]  43.7% of 71.05MiB at 982.12KiB/s ETA 00:41
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 99.67KiB/s, downloaded 2848KiB before throttling)
[download]  47.6% of 71.05MiB at 999.42KiB/s ETA 00:38
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 54.76KiB/s, downloaded 2848KiB before throttling)
[download]  51.5% of 71.05MiB at 1000.74KiB/s ETA 00:35
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 105.12KiB/s, downloaded 2848KiB before throttling)
[download]  55.5% of 71.05MiB at 1009.74KiB/s ETA 00:32
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 90.60KiB/s, downloaded 2848KiB before throttling)
[download]  59.3% of 71.05MiB at 1021.74KiB/s ETA 00:28
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 122.39KiB/s, downloaded 2816KiB before throttling)
[download]  63.2% of 71.05MiB at  1.01MiB/s ETA 00:25
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 75.48KiB/s, downloaded 2848KiB before throttling)
[download]  65.3% of 71.05MiB at  1.00MiB/s ETA 00:24
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 381.17KiB/s, downloaded 1472KiB before throttling)
[download]  69.1% of 71.05MiB at  1.02MiB/s ETA 00:21
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 316.73KiB/s, downloaded 2816KiB before throttling)
[download]  70.9% of 71.05MiB at  1.02MiB/s ETA 00:20
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 332.67KiB/s, downloaded 1248KiB before throttling)
[download]  70.9% of 71.05MiB at  1.01MiB/s ETA 00:20
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 361.57KiB/s, downloaded 64KiB before throttling)
[download]  74.9% of 71.05MiB at  1.01MiB/s ETA 00:17
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 51.19KiB/s, downloaded 2880KiB before throttling)
[download]  75.4% of 71.05MiB at 1017.96KiB/s ETA 00:17
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 378.02KiB/s, downloaded 384KiB before throttling)
[download]  79.4% of 71.05MiB at 1015.14KiB/s ETA 00:14
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 118.29KiB/s, downloaded 2880KiB before throttling)
[download]  79.7% of 71.05MiB at 1006.10KiB/s ETA 00:14
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 386.13KiB/s, downloaded 224KiB before throttling)
[download]  83.7% of 71.05MiB at 999.06KiB/s ETA 00:11
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 50.03KiB/s, downloaded 2912KiB before throttling)
[download]  87.7% of 71.05MiB at 978.31KiB/s ETA 00:09
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 51.73KiB/s, downloaded 2912KiB before throttling)
[download]  91.6% of 71.05MiB at 966.09KiB/s ETA 00:06
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 403.69KiB/s, downloaded 2816KiB before throttling)
[download]  95.5% of 71.05MiB at 964.43KiB/s ETA 00:03
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 240.86KiB/s, downloaded 2880KiB before throttling)
[download]  99.4% of 71.05MiB at 964.62KiB/s ETA 00:00
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 152.70KiB/s, downloaded 2848KiB before throttling)
[download] 100% of 71.05MiB in 01:15

[dstftw]

Breaks downloading completely:

PS C:\dev\youtube-dl\master> py -2.7 .\youtube_dl\__main__.py -v BaW_jenozKc
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'BaW_jenozKc']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2016.12.15
[debug] Git HEAD: 4fee44d
[debug] Python version 2.7.12 - Windows-10-10.0.10240
[debug] exe versions: ffmpeg N-81192-g04da20e, ffprobe N-81192-g04da20e, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] BaW_jenozKc: Downloading webpage
[youtube] BaW_jenozKc: Downloading video info webpage
[youtube] BaW_jenozKc: Extracting video information
[youtube] BaW_jenozKc: Downloading MPD manifest
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[debug] Invoking downloader on u'https://r2---sn-ug5onuxaxjvh-c5me.googlevideo.com/videoplayback?id=05a5bf8de9e8cca7&itag=137&source=youtube&requiressl=yes&mv=m&ms=au&pl=21&initcwndbps=3367500&mn=sn-ug5onuxaxjvh-c5me&mm=31&ratebypass=yes&mime=video/mp4&gir=yes&clen=2208750&lmt=1387961822987808&dur=9.800&key=dg_yt0&mt=1482062833&upn=0iq45Go0Fi0&signature=78D766DDE5C76DAFDA3174E7E3765B59F2149A7F.58FDA990D2FA0D507EE4AF9AD0E1AD5CBFA5B554&ip=<snip>&ipbits=0&expire=1482084789&sparams=ip,ipbits,expire,id,itag,source,requiressl,mv,ms,pl,initcwndbps,mn,mm,ratebypass,mime,gir,clen,lmt,dur'
[download] Destination: youtube-dl test video ''_-BaW_jenozKc.f137.mp4
[download]   0.0% of 2.11MiB at 111.11KiB/s ETA 00:19Traceback (most recent call last):
  File ".\youtube_dl\__main__.py", line 19, in <module>
    youtube_dl.main()
  File "C:\dev\youtube-dl\master\youtube_dl\__init__.py", line 445, in main
    _real_main(argv)
  File "C:\dev\youtube-dl\master\youtube_dl\__init__.py", line 435, in _real_main
    retcode = ydl.download(all_urls)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1791, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 705, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 751, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1435, in process_video_result
    self.process_info(new_info)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1693, in process_info
    partial_success = dl(fname, new_info)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1641, in dl
    return fd.download(name, info)
  File "C:\dev\youtube-dl\master\youtube_dl\downloader\common.py", line 353, in download
    return self.real_download(filename, info_dict)
  File "C:\dev\youtube-dl\master\youtube_dl\downloader\http.py", line 236, in real_download
    block_rate = block_size / (time.time() - block_start)
ZeroDivisionError: float division by zero

[Serkora]

Right, all that time thinking about how to deal with slow internet and got rekt by a too fast one. Fixed, you can try again.

[dstftw]

> py -2.7 .\youtube_dl\__main__.py -v http://www.pornhub.com/view_video.php?viewkey=648719015 --proxy socks5://127.0.0.1:9050 --avoid-throttling
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'http://www.pornhub.com/view_video.php?viewkey=648719015', u'--proxy', u'socks5://127.0.0.1:9050', u'--avoid-throttling']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2016.12.31
[debug] Git HEAD: ab70f89
[debug] Python version 2.7.12 - Windows-10-10.0.10240
[debug] exe versions: ffmpeg N-81192-g04da20e, ffprobe N-81192-g04da20e, rtmpdump 2.4
[debug] Proxy map: {u'http': u'socks5://127.0.0.1:9050', u'https': u'socks5://127.0.0.1:9050'}
[PornHub] 648719015: Downloading webpage
[debug] Invoking downloader on u'http://cdn-d-vid-public.pornhub.com/videos/201306/28/14084201/vl_720_1073k_14084201.mp4?ttl=1483361684&ri=1228800&rs=1344&hash=cbb18edf1250c8f2e87993d87aca75f2'
[download] Resuming download at byte 20780290
[download] Destination: Seductive Indian beauty strips down and fingers her pink pussy-648719015.mp4
[download]  45.0% of 47.31MiB at 663.43KiB/s ETA 00:40
[throttling] Throttling started or is continuing, block rate = 160.45KiB/s, peak rate = 663.43KiB/s
[throttling] Throttling detected! peak rate = 663.43KiB/s, current rate = 80.20KiB/s, setting threshold to 0.34 and block size limit to 64KiB
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 663.43KiB/s, block rate = 160.45KiB/s, downloaded 2004KiB before throttling)
Traceback (most recent call last):
  File ".\youtube_dl\__main__.py", line 19, in <module>
    youtube_dl.main()
  File "C:\dev\youtube-dl\master\youtube_dl\__init__.py", line 445, in main
    _real_main(argv)
  File "C:\dev\youtube-dl\master\youtube_dl\__init__.py", line 435, in _real_main
    retcode = ydl.download(all_urls)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1791, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 705, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 751, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1435, in process_video_result
    self.process_info(new_info)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1699, in process_info
    success = dl(filename, info_dict)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1641, in dl
    return fd.download(name, info)
  File "C:\dev\youtube-dl\master\youtube_dl\downloader\common.py", line 353, in download
    return self.real_download(filename, info_dict)
  File "C:\dev\youtube-dl\master\youtube_dl\downloader\http.py", line 334, in real_download
    data, throttled = self.speed_up(data, request, peak_rate, block_rate, byte_counter, throttling_threshold)
  File "C:\dev\youtube-dl\master\youtube_dl\downloader\http.py", line 43, in speed_up
    request = sanitized_Request(request.full_url, None, request.headers)
  File "C:\Python\Python27\lib\urllib2.py", line 254, in __getattr__
    raise AttributeError, attr
AttributeError: full_url

[Serkora]

Sorry, really inconvenient for me to test on various python versions right now. (Though, for changes in such a core file I probably should have).

[rodcul]

Hey @Serkora any chance you could get this working? I'm trying to download over 125 tracks from Mixcloud and it would be REALLY useful...

[Serkora]

@rodcul Is it not working for you right now? What kind of errors are you seeing etc? I'm not aware of any other issues, but would gladly solve them. I'm just waiting for dstftw to get around this request again to decide what to do with it.

[Gunny123]

Any update on this issue?

[Serkora]

Hi @dstftw , Is this thing rejected and abandoned or there's a chance it might be integrated? I'm happy to fix/improve whatever's required, just need some feedback.

[zannix]

I'm eager to see this working, especially now that YouTube is throttling download speed. It's really important.

[mzso]

@zannix commented on 2018. jan. 18. 09:59 CET:

I'm eager to see this working, especially now that YouTube is throttling download speed. It's really important.

I noticed downloads being really slow 1-2 days ago. But coincidentally I upgraded yt-dl not much before it. So it's youtube that started doing things differently and not yt-dl?

[zannix]

Yes, see the references a few posts above. This functionality would really be priceless right now.

[arunesh90]

@yan12125 bit weird that you refer people to here for a "possible solution" while the creator of the PR had asked a collaborator for feedback but got ignored

[LikeLakers2]

What bothers me about this issue (and excuse me for bringing it here, the discussion issue that was linked #6923 was locked so I'm forced to comment this here) is that, for me at least, removing ffmpeg from the same folder as youtube-dl seems to fix the throttling. With ffmpeg in the directory, I get horribly slow download speeds from YouTube, but the moment I remove it, the video downloads go insanely quick.

While I'm not going to pin the blame on anything here (I haven't delved into the code of anything involved), this just doesn't feel right. If we're using ffmpeg to download when it's available (that's an if since I'm not sure on how ydl works), has anyone checked to make sure we couldn't implement this sort of fix into ffmpeg instead?

[Hrxn]

^ Keep ffmpeg in the same folder as youtube-dl and do a test like this: youtube-dl -v --ignore-config -f best <YT-URL>

Expected if it is also fast. No rate limit for non-DASH formats, but all you get is 720p this way, not more.

[mzso]

@Hrxn commented on 2018. jan. 24. 00:36 CET:

^

Keep ffmpeg in the same folder as youtube-dl and do a test like this:

youtube-dl -v --ignore-config -f best <YT-URL>

Expected if it is also fast.

No rate limit for non-DASH formats, but all you get is 720p this way, not more.

I wonder if they just didn't get to classic streams it or they just don't intend to. (It's certainly possible to rate control them. Other websites did it several years ago)

[Gunny123]

@mzso I just downloaded a personal 2 minute upload of mine from YouTube and I was getting a near constant 4.75 MiB/s off of my gigabit internet connection.

It seems mileage may vary heavily off what video one is downloading, etc.

Other sites like PH, throttle downloads to 250 KiB/s or less which can be brutal. Although it seems their compression has a way of crunching file sizes down to a quarter or more of the original file size which makes it tolerable.

[Hrxn]

If this was your own uploaded video, did you download with user credentials or .netrc?

[Gunny123]

@Hrxn I just popped the link into my terminal and downloaded I didn't use any credentials or .netrc. Can you elaborate on those? I have no idea what those really are lol

[mzso]

@Gunny123 commented on 2018. jan. 25. 14:24 CET:

@mzso I just downloaded a personal 2 minute upload of mine from YouTube and I was getting a near constant 4.75 MiB/s off of my gigabit internet connection.

It seems mileage may vary heavily off what video one is downloading, etc.

Other sites like PH, throttle downloads to 250 KiB/s or less which can be brutal. Although it seems there compression has a way of crunching file sizes down to a quarter or more of the original file size.

I did a test it seems like non-dash stuff are much less throttled. dash webm 720p downloaded <1 MB/s Non-dash 720p mp4 downloaded at first around 40-50MB/s then it got throttled to 25MB/s.

[Hrxn]

@Gunny123 Yeah, because it might be possible that there is a difference between using youtube-dl to sign in to your account on YouTube (Can be done with --username and --password or by using these in a .netrc file) and using youtube-dl without any authentication, what you apparently did. You might be right that this is dependent on the video you actually try to download, or that the rate limiting done by YouTube is pretty random in general.

@mzso Yes, there is a difference between DASH and non-DASH formats. I already mentioned this in here.

[mzso]

@Hrxn commented on 2018. jan. 25. 16:06 CET:

@mzso Yes, there is a difference between DASH and non-DASH formats. I already mentioned this in here.

My point was that non-dash is also throttled. (But not to the same horrible degree)

[pickae]

@mzso I would say from experience that the h264 and AAC formats where always somewhat throttled on youtube because they are delivered in many small fragments instead of one big file. It would annoy me slightly when a channel wasn't available in VP9 and opus because that means it would be slower and worse in terms of quality and quality per filesize.

VP9 and opus used to saturate 100mbit/s all the time. Now they are throttled even worse than the old formats. But going back to h264 is not a great solution as the quality is just lower.

aria2c as an external downloader brings VP9 and opus downloads back to speed though.

[mzso]

@pickae commented on 2018. jan. 25. 21:45 CET:

@mzso I would say from experience that the h264 and AAC formats where always somewhat throttled on youtube because they are delivered in many small fragments instead of one big file. It would annoy me slightly when a channel wasn't available in VP9 and opus because that means it would be slower and worse in terms of quality and quality per filesize.

VP9 and opus used to saturate 100mbit/s all the time. Now they are throttled even worse than the old formats. But going back to h264 is not a great solution as the quality is just lower.

aria2c as an external downloader brings VP9 and opus downloads back to speed though.

I don't see to much of a difference. Do I need to configure something in aria2c? How?

[mzso]

@pickae commented on 2018. jan. 25. 21:45 CET:

aria2c as an external downloader brings VP9 and opus downloads back to speed though.

Well for whatever reason, the options to increase connections don't work...

[pickae]

@mzso I literally just installed it for that purpose and didn't configure anything in aria2. First without the external downloader:

youtube-dl -f 308 https://www.youtube.com/watch?v=1La4QzGeaaQ
[youtube] 1La4QzGeaaQ: Downloading webpage
[youtube] 1La4QzGeaaQ: Downloading video info webpage
[youtube] 1La4QzGeaaQ: Extracting video information
[download] Destination: Peru 8K HDR 60FPS (FUHD)-1La4QzGeaaQ.webm
[download] 100% of 489.65MiB in 03:13

Now with aria2c

youtube-dl --external-downloader aria2c -f 308 https://www.youtube.com/watch?v=1La4QzGeaaQ
[youtube] 1La4QzGeaaQ: Downloading webpage
[youtube] 1La4QzGeaaQ: Downloading video info webpage
[youtube] 1La4QzGeaaQ: Extracting video information
[download] Destination: Peru 8K HDR 60FPS (FUHD)-1La4QzGeaaQ.webm

01/25 22:56:06 [NOTICE] Downloading 1 item(s)
[#3cee2c 483MiB/489MiB(98%) CN:4 DL:8.1MiB]                                                                      
01/25 22:57:00 [NOTICE] Download complete: /home/Peru 8K HDR 60FPS (FUHD)-1La4QzGeaaQ.webm.part

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
3cee2c|OK  |   9.0MiB/s|/home/Peru 8K HDR 60FPS (FUHD)-1La4QzGeaaQ.webm.part

Status Legend:
(OK):download completed.
[aria2c] Downloaded 513433761 bytes
[download] 100% of 489.65MiB

It's not as good as before where it would simply saturate my 100mbit/s for any selected resolution, but it's viable like this.

The difference is less pronounced when you select crazy formats (like -f 272 for 8K with 60fps) because then even the throttled version allows you decent download speeds (as you would need them while watching in the browser). Conversely, selecting lower resolutions like -f 244 for 480p VP9 only makes the throttling tighter, the download without aria2c takes roughly the same amount of time as for 1080p or higher. aria2c helps at such low resolutions, but it doesn't saturate my 100mbit/s as it does with the 1440p example I posted above.

The difference is more pronounced if you download audio only in opus for long videos.

[mzso]

@h1z1 commented on 2018. jan. 26. 12:59 CET:

Don't know about everywhere else but in the last month or so youtube has enabled some serious throttling. Maybe as a political statement for net neutrality.. Serious as in Gigabit connection yet can't pull more then a mb/s or less, it's to the point videos actually buffer again which hasn't happened in many years.

Regarding aria2, they dropped the defaults such that it's kinda pointless for most videos from youtube. The wording in their help is quite misleading since the parallel split is 5 but per server max is still 1, (split size 20M). In essence that means you'll only benefit from it if the file you're downloading is at least 40M AND you increase the per host max (-x).

But why don't the settings to increase connections work? (--split; --min-split-size)

[Baddest-Juju]

@h1z1 Does axel really speed up downloads or is that just a placebo effect? I tried aria2c and it didn't make a difference, axel doesn't have a windows binary to my knowledge.

[Gunny123]

Can someone provide a ELI5 as to why this issue/PR is difficult to merge? Excuse my blatant ignorance of internet protocols and lack of understanding in terminal.

[Hrxn]

Because that is a somewhat old PR with a generic method against throttling for standard HTTP, i.e. by repeating requests continually. But this requires Content-Range Header support, which is often not the case. And YouTube's DASH is another story again.