youtube-dl
youtube-dl copied to clipboard
ytdl-org
ITV Hub JSON error
Checklist
- [x] I'm reporting a broken site support
- [x] I've verified that I'm running youtube-dl version 2021.04.26
- [x] I've checked that all provided URLs are alive and playable in a browser
- [x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [x] I've searched the bugtracker for similar issues including closed ones
Verbose log
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.itv.com/hub/tipping-point/2a1875a1123', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2021.04.26
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: none
[debug] Proxy map: {}
[debug] Using fake IP 25.161.160.85 (GB) as X-Forwarded-For.
[ITV] 2a1875a1123: Downloading webpage
[ITV] 2a1875a1123: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpeav8wi2f\build\youtube_dl\extractor\common.py", line 634, in _request_webpage
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpeav8wi2f\build\youtube_dl\YoutubeDL.py", line 2279, in urlopen
File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default
Description
Several programmes on ITV Hub now fail to download. URL's such as
https://www.itv.com/hub/the-chase/1a7842a1761 https://www.itv.com/hub/tipping-point/2a1875a1123 https://www.itv.com/hub/tenable/10a0805a0021
fail to download though some like
https://www.itv.com/hub/lorraine/1a9360a2830
continue to work
Same here, hopefully it's not DRM.
[debug] System config: []
[debug] User config: ['-o', 'F:/Temp/youtubedl/%(title)s.%(ext)s']
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.itv.com/hub/heartbeat/Ya0757a0043']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2021.04.26
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg N-95216-ge6625ca41f, ffprobe N-95216-ge6625ca41f, rtmpdump 2.4
[debug] Proxy map: {}
[debug] Using fake IP 25.151.6.58 (GB) as X-Forwarded-For.
[ITV] Ya0757a0043: Downloading webpage
[ITV] Ya0757a0043: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpeav8wi2f\build\youtube_dl\extractor\common.py", line 634, in _request_webpage
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpeav8wi2f\build\youtube_dl\YoutubeDL.py", line 2279, in urlopen
File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default
I first came across this problem on the 28th April when trying to download the ITV news programmes for that day. Other programmes prior to the 28th April I haven't had a problem with the completion of the download. But then programmes such as,
youtube-dl https://www.itv.com/hub/good-morning-britain/2a3211a3033 youtube-dl https://www.itv.com/hub/lorraine/1a9360a2831 youtube-dl https://www.itv.com/hub/this-morning/2a6292a0581 for the 28th April as well comes up with the error & doesn't start downloading at all as well.
C:\Users\User\Videos\youtube-dl Downloads>youtube-dl https://www.itv.com/hub/lorraine/1a9360a2831 --verbose [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['https://www.itv.com/hub/lorraine/1a9360a2831', '--verbose'] [debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252 [debug] youtube-dl version 2021.04.26 [debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041 [debug] exe versions: ffmpeg 4.0.2, ffprobe 3.4.1 [debug] Proxy map: {} [debug] Using fake IP 25.168.96.112 (GB) as X-Forwarded-For. [ITV] 1a9360a2831: Downloading webpage [ITV] 1a9360a2831: Downloading JSON metadata ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpeav8wi2f\build\youtube_dl\extractor\common.py", line 634, in _request_webpage File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpeav8wi2f\build\youtube_dl\YoutubeDL.py", line 2279, in urlopen File "C:\Python\Python34\lib\urllib\request.py", line 470, in open File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response File "C:\Python\Python34\lib\urllib\request.py", line 508, in error File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default
C:\Users\User\Videos\youtube-dl Downloads>
Luckily at the moment, STV player still works, but anything from ITV2, 3, 4 or CITV would not be downloadable because STV is only the Channel 3 contractor for Scotland. And you have to hope they haven't opted out of the ITV schedule and are showing the show you want.
Same here, hopefully it's not DRM.
500 Internal Server Errors are usually temporary. It means that for some reason, the server was unable to process the request that was sent to it.
If it was DRM related, I would not expect a 500 error.
Someone suggested in the other thread that if you kept trying, the programme would eventually download. That works for 403 Forbidden errors but doesn't appear to work for this.
The problem seems to be that for some programmes the server is just finding it impossible to send the JSON playlist back which means that YT-DL has no way to attempt to download the programme. Why this only seems to affect some programmes and not others though is a bit of a mystery.
I've been noticing the same thing over the past day or two also. Here's another example of an identical failure. It doesn't appear retrying works, just flat out refuses to download however many times over a 12 hour window. Running same versions as @TIGHazard1. And, as reported, some downloads do work, just the ones that don't never seem to work. Have also tried with --no-geo-bypass giving identical failure. I am UK based.
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.itv.com/hub/viewpoint/10a0556a0003']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2021.04.26
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg git-2019-11-01-53c21c2, ffprobe git-2019-11-01-53c21c2
[debug] Proxy map: {}
[debug] Using fake IP 25.41.255.169 (GB) as X-Forwarded-For.
[ITV] 10a0556a0003: Downloading webpage
[ITV] 10a0556a0003: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpeav8wi2f\build\youtube_dl\extractor\common.py", line 634, in _request_webpage
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpeav8wi2f\build\youtube_dl\YoutubeDL.py", line 2279, in urlopen
File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default
this comment applies to the two issues here and here too, can someone please see whether physical old ios devices are failing on these videos? if so and if itv are still supporting them then can be reported as a bug
I don't own any ios devices so sadly can't test that. Is anyone else able to?
I agree that many programs on ITV Hub now fail to download.
On 28th April I was able to download URL's
https://www.itv.com/hub/viewpoint/10a0556a0001 https://www.itv.com/hub/viewpoint/10a0556a0002
On 29th April I was unable to download URL
https://www.itv.com/hub/viewpoint/10a0556a0003
Moreover, if I now try to download the first two above URL's they now fail with the same error message that others are reporting, namely:
[ITV] 2a7855a0082: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
So ITV have changed something server side. I don't think it's DRM (yet), but maybe a missing header or auth token?
So ITV have changed something server side. I don't think it's DRM (yet), but maybe a missing header or auth token?
I cannot see a 500 error being DRM related. The error shows before the download has even been attempted. If DRM made the videos impossible to download then they would fail with some sort of unable to download error not a server error.
@gewyvoso I can confirm you get the http 500 error from your first link when I try yt-dl on OSX. I can also confirm that I can access the link from the itv hub with an old iphone using ios. In both cases using a VPN. Is this the info that you want?
@dstftw Sorry but geo-restriction isn't the issue in this case. Unless there is something we have missed? That seems like an unfair label to have put on it. We can all stream direct from the website but get errors from yt-dl.
@gewyvoso I can also confirm I have tried it on an iPad with no issues. Which is annoying.
@Vangelis66 Any ideas what might be happening here?
@dstftw I tend to agree with @torbs723 that this isn't geo-restriction, it's something else.
is anyone able to set up man in the middle to view the traffic on working old ios? perhaps it's working in a different way from itv.py
@dstftw, giving a geo tag to this is off topic, not the above two comments. one of my two linked threads gives detailed info about geo behaviour and that is different from what is occurring here. could be that it is more and different geo restriction and that you know something about it, so then please tell us that
@LFS1 & @torbs723, i was surprised it did work. thank you for doing. that was one logical avenue to investigate. like i said, maybe the ytdl plugin is working differently from what itv are doing for real. if so, that could be a fix
all this is academic to me, i'm not fussed, although i have heavily cached itv shows using this hls method til now. is there a place where people discuss the workings of these plugins? people seem to present good knowledge in these threads, which seems to have been picked up elsewhere. i would like to chat there, this forum here does not seem right for it
@gewyvoso Thank you for the support
i was surprised it did work. thank you for doing. that was one logical avenue to investigate. like i said, maybe the ytdl plugin is working differently from what itv are doing for real. if so, that could be a fix
It is possible though that the streams we are having problems with are used by other devices possibly old old iPhones, iPads etc that maybe don't support ITV's app?
this is what i mean. looking at the plugin code, it was like it was magicked into being. there was no explanation of its workings except by reading between the lines. happy to do that though, but what i would like to see is where discussions took place leading to that. that is where i can contribute and would like to, not here. until then i find myself holding back my thoughts . . . . thanks for discussing though. i am well interested in this kind of thing. there are people of many different mindsets here. good to have a more focussed group of like minds on it. then can talk freely. do you know what i mean?
(28934 is another one. looks like i commented there same minute you commented at 28939)
If only people would spend 30 seconds doing a search before they post a new issue.
Same problem here, but for me it only happens with videos that have appeared in the last day or two. I can still D/L older videos. For example, https://www.itv.com/hub/robin-of-sherwood/ROBa03a010, from 28 April, can be downloaded OK, but https://www.itv.com/hub/robin-of-sherwood/ROBa03a011, from 29 April, and https://www.itv.com/hub/robin-of-sherwood/ROBa03a012, from 30 April, both fail with the HTTP Error 500 with latest youtube-dl version 2021.04.26. e.g.
[ITV] ROBa03a011: Downloading webpage [ITV] ROBa03a011: Downloading JSON metadata ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 634, in _request_webpage return self._downloader.urlopen(url_or_request) File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2279, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "/usr/lib/python2.7/urllib2.py", line 435, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 548, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.7/urllib2.py", line 473, in error return self._call_chain(*args) File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain result = func(*args) File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) `
@torbs723 wrote:
@Vangelis66 Any ideas what might be happening here?
I was tagged (in a now hidden comment) to troubleshoot this... 😄
It is now already Orthodox Easter Sunday here, I won't be available for the most part of the next two days... However, since I, too, have a vested interest in ITVHub shows fetched via yt-dl, I took a stab at it...
But first, let's discuss the geo-restricted label attributed to this issue...
In general, it is not a good idea to aggravate the main developer(s) of yt-dl 😉 ; perhaps the label was just put there to highlight the fact that for anyone to even start troubleshooting this, an ITV Hub whitelisted IP address is required; sometimes, a generic UK IP won't cut it, you need an "ITV zone" one (e.g. excluding Scotland) ...
My analysis has indeed proven (to me, at least, but in accordance to comments by other people here) that this new ITV HUB issue is not related to geo-fencing...
While I am discussing geo-fencing, the ITV Hub people, some time in the month of March (2021), imposed even stricter geo-blocking schemes... 💢
For many years, their Playlist APIs (= the ones that yield stream data) would only enforce a "soft" geo-block, that could be circumvented via an X-Forwarded-For:UK IP request header; if you look closely at logs people have posted, it's line
[debug] Using fake IP 25.161.160.85 (GB) as X-Forwarded-For.
From mid-March, if you happen to be overseas, you need a whitelisted UK HTTPS/SOCKS proxy/UK VPN to even get the master HLS playlist URI ("Downloading JSON metadata" in yt-dl's logs); of course, the HLS stream itself is fully geo-blocked, you have to use the same "means" to dump if overseas... Again, this new "issue" isn't related to stricter geo-fencing practices imposed last March (as it also occurs for people within the UK...).
It well appears that the ITV Hub people made some changes during these last days, certainly during last week, that make recently uploaded ITV hub shows inaccessible to yt-dl; case in point is the Jonathan Ross show:
Last week's episode
youtube-dl -F https://www.itv.com/hub/the-jonathan-ross-show/2a1166a0204
[ITV] 2a1166a0204: Downloading webpage
[ITV] 2a1166a0204: Downloading JSON metadata
[ITV] 2a1166a0204: Downloading m3u8 information
[info] Available formats for 2a1166a0204:
format code extension resolution note
hls-105 mp4 audio only 105k , mp4a.40.2@ 98k
hls-565 mp4 512x288 565k , avc1.64001F@ 434k, mp4a.40.2@ 98k
hls-762 mp4 512x288 762k , avc1.64001F@ 620k, mp4a.40.2@ 98k
hls-1058 mp4 896x504 1058k , avc1.64001F@ 899k, mp4a.40.2@ 98k
hls-1344 mp4 896x504 1344k , avc1.64001F@1169k, mp4a.40.2@ 98k
hls-1633 mp4 896x504 1633k , avc1.64001F@1442k, mp4a.40.2@ 98k (be
st)
Yerterday's episode:
youtube-dl -F https://www.itv.com/hub/the-jonathan-ross-show/2a1166a0205
[ITV] 2a1166a0205: Downloading webpage
[ITV] 2a1166a0205: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error (
caused by <HTTPError 500: 'Internal Server Error'>); please report this issue on
https://yt-dl.org/bug . Make sure you are using the latest version; type youtu
be-dl -U to update. Be sure to call youtube-dl with the --verbose flag and incl
ude its complete output.
(the test was done using the very same connection/IP).
Now, my deeper analysis was done focusing on Viewpoint-SE01EP03 (sadly, the whole series is due to expire imminently): https://www.itv.com/hub/viewpoint/10a0556a0003 As most of you may know already, desktop browsers are being served MPEG-DASH streams with Common Encryption (cenc=DRM), the Google-owned WidevineCDM is required for playback 👎 💢 ; yt-dl can't/won't handle cenc, instead it has to resort to HLSe (AES-128 encrypted) streams, targeting older iOS devices...
So the very first thing I did was to emulate such a device in my browser, by setting up a site-specific-user-agent-override (SSUAO) for itv.com; depending on browser used, this can be achieved via an about:config setting (Firefox derived browsers) or, more commonly, via an extension (Chromium browsers) that spoofs your user-agent; please, don't ask for instructions/how-to here, it isn't the appropriate place... I chose to use the iPhone5 UA below:
Mozilla/5.0 (iPhone; CPU iPhone OS 5_0 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A334 Safari/7534.48.3
Of course, ITVHub requires you to be logged-in to send any stream data to your browser (this hasn't been yet a requirement for yt-dl 👍 ); being logged in and with the above "mobile" UA, I inspected browser's Web Console (CTRL+SHIFT+K) and I clearly saw the POST XHR request to their playlist API:
https://magni.itv.com/playlist/itvonline/ITV/10_0556_0003.001
(don't try this directly with a GET request), the response of which I am attaching here: POST-ITV.json.txt
BTW, this is equivalent to yt-dl "Downloading JSON metadata"; as one can see, no HTTP Error 500: Internal Server Error was generated in my test; this rules out intermittent server issues on the ITV side...
From that (see attached) JSON response, I extracted the master HLSe playlist URI:
https://itvpnpmobile.content.itv.com/10-0556-0003-001/6/2/VAR003/10-0556-0003-001_6_2_VAR003.ism/.m3u8?hdnea=st=1619890241~exp=1619911841~acl=/10-0556-0003-001/*~data=nohubplus~hmac=ec46cb4825a413f0681ce4b94f72c5f82c9fb42fb6d424e746976a5d2c26aa99
(the same URI can be seen and copied from within the browser's Web Console 😉 ), which looks like below:
#EXTM3U
#EXT-X-VERSION:4
## Created with OEM(version=1.7.25)
# AUDIO groups
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio-aacl-128",NAME="audio",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2"
# variants
#EXT-X-STREAM-INF:BANDWIDTH=313000,CODECS="mp4a.40.2,avc1.4D401E",RESOLUTION=512x288,FRAME-RATE=50,AUDIO="audio-aacl-128",CLOSED-CAPTIONS=NONE
10-0556-0003-001_6_2_VAR003-audio=128000-video=167161.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=596000,CODECS="mp4a.40.2,avc1.4D401F",RESOLUTION=512x288,FRAME-RATE=50,AUDIO="audio-aacl-128",CLOSED-CAPTIONS=NONE
10-0556-0003-001_6_2_VAR003-audio=128000-video=433735.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=794000,CODECS="mp4a.40.2,avc1.4D401F",RESOLUTION=512x288,FRAME-RATE=50,AUDIO="audio-aacl-128",CLOSED-CAPTIONS=NONE
10-0556-0003-001_6_2_VAR003-audio=128000-video=620155.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1064000,CODECS="mp4a.40.2,avc1.4D401F",RESOLUTION=896x504,FRAME-RATE=50,AUDIO="audio-aacl-128",CLOSED-CAPTIONS=NONE
10-0556-0003-001_6_2_VAR003-audio=128000-video=875433.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1655000,CODECS="mp4a.40.2,avc1.4D401F",RESOLUTION=896x504,FRAME-RATE=50,AUDIO="audio-aacl-128",CLOSED-CAPTIONS=NONE
10-0556-0003-001_6_2_VAR003-audio=128000-video=1432734.m3u8
# variants
#EXT-X-STREAM-INF:BANDWIDTH=136000,CODECS="mp4a.40.2",AUDIO="audio-aacl-128"
10-0556-0003-001_6_2_VAR003-audio=128000.m3u8
# keyframes
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=23000,CODECS="avc1.4D401E",RESOLUTION=512x288,URI="keyframes/10-0556-0003-001_6_2_VAR003-video=167161.m3u8"
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=116000,CODECS="avc1.4D401F",RESOLUTION=896x504,URI="keyframes/10-0556-0003-001_6_2_VAR003-video=875433.m3u8"
This URI is short-lived, you have to act swiftly before its auth-token expires... Feeding that directly to yt-dl, I can initiate a successful download:
youtube-dl --console-title --hls-prefer-native --hls-use-mpegts -c --fixup never "https://itvpnpmobile.content.itv.com/10-0556-0003-001/6/2/VAR003/10-0556-0003-001_6_2_VAR003.ism/.m3u8?hdnea=st=1619890241~exp=1619911841~acl=/10-0556-0003-001/*~data=nohubplus~hmac=ec46cb4825a413f0681ce4b94f72c5f82c9fb42fb6d424e746976a5d2c26aa99" -o "ITV_Viewpoint_Se1Ep3_202104282100[10a0556a0003].ts" =>
[generic] *~data=nohubplus~hmac=ec46cb4825a413f0681ce4b94f72c5f82c9fb42fb6d424e7
46976a5d2c26aa99: Requesting header
[generic] *~data=nohubplus~hmac=ec46cb4825a413f0681ce4b94f72c5f82c9fb42fb6d424e7
46976a5d2c26aa99: Downloading m3u8 information
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 461
[download] Destination: ITV_Viewpoint_Se1Ep3_202104282100[10a0556a0003].ts
[download] 2.6% of ~543.70MiB at 942.18KiB/s ETA 12:38
I did spend a lot of time trying to emulate the above POST XHR my browser sent in a yt-dl command, by specifying all needed request headers (referer/user-agent/cookies etc.), I even tried the cookies.txt method, sadly all to no avail 😭 ... It's a real pity that I can't code (in Python or other language), I hope my findings can be put to good use by actual coders...
In conclusion: ITV staff have implemented in recent days "changes" (I wouldn't rule out this having been done deliberately to thwart downloaders) that would render current itvIE useless once old shows expire, perhaps sooner if these changes propagate to all ITVhub content... At the time of this writing, this isn't related to an implementation of pure DRM (e.g. Apple FairPlay), nor to further changes to their geo-fencing practices... Therefore, I humbly concluded it's a genuine itv.py bug that needs to be addressed in code... @remitamine, you seem to be the defacto maintainer of the itvIE, please, when your schedules permit, could you take a deeper look into this? Also pinging @dirkf who happens to be UK-based, a Python coder and quite knowledgeable 😄 in UK media sites...
My best wishes to all, my deepest thanks to the yt-dl dev team 👍 , Happy Easter Sunday to all those that observe it today... 😃
@Vangelis66 As always Vangelis, thank you so much for your detailed analysis of this issue. And thank you for your confirmation that this doesn't relate to DRM.
Really appreciate the analysis @Vangelis66. Learnt something about spoofing the user agent as well. Taking it from here is beyond me but hopefuly the coding heroes can find some time and motivation to pick it up. Happy Easter Sunday.
I too have just learnt about spoofing user agents and can concur with @Vangelis66 that downloads from ITV (UK based) are still possible as described above. Happy Easter.
I asked a friend of mine in London to test yt-dl again fridays episode of the chase. He has confirmed that the same 500 error occurs irrespective of where you are located. I hope this helps.
I asked a friend of mine in London to test yt-dl again fridays episode of the chase. He has confirmed that the same 500 error occurs irrespective of where you are located. I hope this helps.
But is available with the @Vangelis66 method
OK, I think I got to the bottom of it... 🎉
My M.O. was comparing the Page Source of ITV Hub shows that work with latest yt-dl, e.g.
(previous) "The Jonathan Ross Show -SE17E03":
https://www.itv.com/hub/the-jonathan-ross-show/2a1166a0204
with the ones that are now "broken", e.g. (latest) "The Jonathan Ross Show -SE17E04":
https://www.itv.com/hub/the-jonathan-ross-show/2a1166a0205
The point of comparison was the "data-video-variants" block; in the first case (working), it looks like below (pretty-printed, for clarity):
data-video-variants="{
'dotcom':[
['hls','fairplay'],
['mpeg-dash','playready','outband-webvtt'],
['hls','fairplay','outband-webvtt'],
['hls','aes','outband-webvtt'],
['hls','aes'],
['mpeg-dash','widevine'],
['mpeg-dash','clearkey','outband-webvtt'],
['mpeg-dash','widevine','outband-webvtt'],
['mpeg-dash','playready'],
['mpeg-dash','clearkey']
],
'mobile':[
['mpeg-dash','clearkey'],
['mpeg-dash','widevine'],
['hls','fairplay'],
['hls','aes'],
['mpeg-dash','widevine','outband-webvtt']
]
}"
In the second case (non-working shows, giving the "HTTP Error 500"), it looks like below:
data-video-variants="{
'dotcom':[
['mpeg-dash','playready','outband-webvtt'],
['hls','fairplay','outband-webvtt'],
['mpeg-dash','widevine'],
['mpeg-dash','widevine','outband-webvtt'],
['hls','fairplay'],
['mpeg-dash','playready']
],
'mobile':[
['mpeg-dash','widevine'],
['mpeg-dash','clearkey'],
['hls','aes'],
['hls','fairplay'],
['mpeg-dash','widevine','outband-webvtt']
]
}"
As you can see, the 'dotcom' group of variants was reduced from 10 to 6 members, the ones that were axed were "softer" encryption methods like (aes | clearkey):
['hls','aes','outband-webvtt'],
['hls','aes'],
['mpeg-dash','clearkey','outband-webvtt'],
['mpeg-dash','clearkey']
which now leaves us with full-fledged DRM (playready | fairplay | widevine) variants within the 'dotcom' group... 😠
As far as itv.py itself is concerned,
https://github.com/ytdl-org/youtube-dl/blob/a0df8a06178e530a1097f177a1faf1d2c609ac99/youtube_dl/extractor/itv.py#L88-L94
the code above instructs it to request a ['hls','aes','outband-webvtt'] variant from within the 'dotcom' group; this variant is now absent in the non-working shows, the PlaylistAPI server can't grant yt-dl's request, hence the
ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error
that started this thread...
As you can see, ITV Hub are going full-steam ahead with fully DRM-ing all their streams; as a most vocal opponent of DRM in every shape or form, I wish them (...) (fill-in the dots yourself, if you get my drift...).
Workaround:
The only remaining yt-dl-compatible video-variant left is ['hls','aes'] inside the 'mobile' group; unfortunately, it doesn't come with the 'outband-webvtt' feature that could be found in the now removed variant ['hls','aes','outband-webvtt'], so subtitles for HLSe streams aren't accessible 👎 ...
I have applied locally a quick-and-dirty patch to, at least, regain access to media streams, by doing in itv.py:
- 'min': ['hls', 'aes', 'outband-webvtt'],
- 'max': ['hls', 'aes', 'outband-webvtt']
+ 'min': ['hls', 'aes'],
+ 'max': ['hls', 'aes']
},
- 'platformTag': 'dotcom'
+ 'platformTag': 'mobile'
... so I can fetch all these programmes currently broken (bound to increase in number, no doubt 😡 ), but without subs for them, even when subs are provided by ITV:
youtube-dl -F https://www.itv.com/hub/the-jonathan-ross-show/2a1166a0205 =>
[ITV] 2a1166a0205: Downloading webpage
[ITV] 2a1166a0205: Downloading JSON metadata
[ITV] 2a1166a0205: Downloading m3u8 information
[info] Available formats for 2a1166a0205:
format code extension resolution note
hls-104 mp4 audio only 104k , mp4a.40.2@ 97k
hls-278 mp4 512x288 278k , avc1.4D401F@ 164k, 25.0fps, mp4a.40.2@ 97k
hls-551 mp4 512x288 551k , avc1.4D401F@ 421k, 25.0fps, mp4a.40.2@ 97k
hls-757 mp4 512x288 757k , avc1.4D401F@ 616k, 25.0fps, mp4a.40.2@ 97k
hls-1031 mp4 896x504 1031k , avc1.4D401F@ 874k, 25.0fps, mp4a.40.2@ 97k
hls-1622 mp4 896x504 1622k , avc1.4D401F@1432k, 25.0fps, mp4a.40.2@ 97k (best)
Obviously, I now need a "real" coder that can harvest the webvtt subtitles from one of the other video-variants (either ['mpeg-dash','widevine','outband-webvtt'] from the 'mobile' group (should be easier?) or from one of the three belonging to the 'dotcom' group); any volunteers? 😄
Annoyingly I have no idea how to compile things if I made that very simple edit. I just download the .exe and run that usually.
Annoyingly I have no idea how to compile things if I made that very simple edit. I just download the .exe and run that usually.
It's easy if your installation unpacks the program into a directory tree containing .py files (typical for Unix-like systems with Python installed separately). Python automatically re-compiles any .py that is newer than its byte-code .pyc counterpart, provided that it's running with sufficient privileges to modify those files. Some builds comprise a single executable from which the program files are extracted at run-time: this discussion may help.
@dirkf Thanks for that. I just download youtube-dl.exe and then run youtube-dl from the command line. I'm not sure where things get unpacked to or anything like that.
@sleaux-meaux Thank you very much for applying this fix.
Impressive work. Really appreciate the efforts from everyone involved in and contributing to this project. Now to patiently wait for the Windows binary to be updated (or figure out how to compile.. wish me luck).