invidious
invidious copied to clipboard
iv-org
Circumventing geo-restrictions
I thought we were already circumventing geo-blocking, but we're not.
Blocked in the U.S.: https://invidio.us/watch?v=xpWu4_hp00M
It would be good to start building a list of videos that are "Blocked in US, Blocked in DE, etc". Do you happen to know in which regions the video linked above is allowed?
Nah, I can't get it to play using any VPN or proxy. Maybe not a great example, as I suppose it's possible that the link is just broken and the geo-restriction aspect isn't really the issue.
Here's an example of a video that is allowed in DE but not US: https://invidio.us/watch?v=rV4Mrl2hQro
It appears that in /videoplayback URLs, for example medium quality for the above video YouTube now returns 403 when the ip parameter does not match the requesting IP.
Helpful link: https://www.youtube.com/get_video_info?video_id=rV4Mrl2hQro
Haven't I seen something like bypassip= params for googlevideo URLs?
You probably saw ratebypass, I don't happen to know what it does.
A few quick tests:
- I misspoke, YouTube only cares about
ip=when the IP is for a video that is not in one of theallowedRegions. - Attempting to access geo-blocked videos through Tor is fine, as long as the exit node is in the correct region.
No, I knew I've seen it. Went through my clipboard manager and found one. IDK if it actually works, but:
https://r3---sn-ab5szn7s.googlevideo.com/videoplayback?fvip=3&source=youtube&signature=230E3DC2DF22B86A40021B84ACAD8839AB515422.0D14359F1455503A0BE6096B7B8B1EFB4CD16927&ipbits=0&pl=18&id=o-AIfZC-oGvCCpSym9jRjuh5PzQ3mlPwK2nLKOpIQb6UFj&dur=180.419&mime=video%2Fmp4&key=cms1&ip=2604%3Aa880%3A400%3Ad1%3A%3A180%3Ad001&ei=fGFjW4_5I5SBhwaVtInwDg&c=WEB&requiressl=yes&sparams=dur,ei,expire,id,ip,ipbits,ipbypass,itag,lmt,mime,mip,mm,mn,ms,mv,pl,ratebypass,requiressl,source&itag=22&lmt=1532956145274494&expire=1533261276&ratebypass=yes&redirect_counter=1&rm=sn-ab5eel7s&req_id=ef8047609366a3ee&cms_redirect=yes&ipbypass=yes&mip=00.000.000.000&mm=31&mn=sn-ab5szn7s&ms=au&mt=1533239832&mv=m
(towards the end)
Doesn't appear to do anything. A while back I put some work into decrypting/resigning /videoplayback URLs but never got anywhere. Notice that in the above URL there is akey and signature, which prevents tampering.
ipbypass=yes seemed like it may be relevant. I was skeptical, but figured it might be worth mentioning.
Working on this right now, currently can get info on the videos linked above. https://invidio.us/watch?v=xpWu4_hp00M appears to be allowed in Germany, Japan, Russia, Thailand (of the regions I've tested). The last thing that needs to be done is to proxy the files themselves, although obviously that will be more bandwidth intensive.
I suggest you to just add this information to the video page, so users can use VPN or proxy to watch videos that are allowed in specific countries.
That's fair enough. I'm already managing downloads for livestreams, and will do the same for #34 and #90, so I thought I might do that here as well. I do plan on adding info to the video page, something like blacklisted countries or whitelisted countries (depending on which is bigger), and that info is already available though the API.
Just pushed fada57a307d66d696d9286fc943c579a3fd22de6, which provides blacklisted/whitelisted countries. I'd take a look at the regions Invidious tries to bypass. It's a pretty small list, but it seems to be able to bypass most of the videos that are geo-restricted. If there are other regions that are allowed by a significant amount of geo-restricted videos, I'll consider adding those regions.
Region codes correspond to those in ISO 3166.
regions that make up a significant amount of traffic
How can you measure that?
I think what I mean is the regions that allow a user to watch the largest amount of content. So Germany, US, JP, etc. I can't measure it very well, but I think there is a metric where some percentage of videos are exclusive to region x, so it should be added to the list that Invidious tries to bypass.
I also just clarified my position a bit: the regions that are allowed by the largest number of geo-restricted videos should be added.
I wonder if its possible to get it to work without having to proxy the video file itself. If it is, then that would be nice.
Attempting to access a geo-blocked file returns a 403 as well, so it appears the only solution is to proxy that as well. This is also an issue for #90: Google will block access to the file itself in some instances, so this is likely the only solution unfortunately.
Not sure if you ever successfully implemented a geo-restriction circumvention. I kinda figured you had bc I hadn't run into any problems in quite a while, but this issue's still open, so perhaps not. Just ran into one though, and besides being geo-blocked, the page takes forever just to load an error message.
https://youtu.be/d9sApXSDtnE
Currently the system in place is to cycle through proxies in specific regions, so most of the videos mentioned work (with dash enabled so videos are also proxied). The reason it's still open is because there's still a lot of room for improvement.. I expect it to be closed alongside #90.
The reason geo-blocked videos can take a long time to load is if Invidious can't find a proxy or if none of the regions Invidious tries return a valid response. I would like to fix that before I close this.
Would it be possible to use Tor for a country from which you dont have a proxy? You can set the exit nodes country if you use tor.
This was suggested in the Matrix server. The issue is that it is not possible to determine where a video is unrestricted without being in an unrestricted region.
The way the current system works is by cycling through proxies in different regions, and finding one where the video is unblocked. Doing this through Tor would put an enormous amount of strain on the network, since it would require building a circuit for every region that is tried.
Also important to note is that video files still have to be proxied from somewhere the video is unblocked. Doing this through Tor is something I'd also rather not do, since it similarly puts strain on the network.
you could reduce cycling if you create statistics about how many video requests are blocked in which country
This is a really annoying issue, lots of video's still don't work for me, especially with music video's like from Vevo, the proxy doesn't seem to fix it for me. even trying alternate instances located closer to me don't work. i'm from Belgium, checking the German, Norwegian, Finnish instance under "alternative instances" doesn't work the one from the Netherlands seems offline, and there's no Belgian instance EDIT: seems like the proxy DOES fix the issue for me, just thought it worked automatically, but seems like you have to enable it in the settings :)
proxy-ing does really cause bandwidth constraints though can't you just catch the video information/url using a proxy, instead of proxy-ing the entire video through invidious servers? that would at least solve the issue that video's don't play, even though they are available in your country when visiting in youtube directly
Sure, it just means you have to track or store the user's region in order to provide correct links. I believe there was also some discussion on this in #460, and I can see it being reasonable to allow a user to "prefer" a different region than where the instance is hosted.
Generally, it's preferable to proxy videos instead of trying to get native links. Mainly because you don't need to manage and retrieve the same video multiple times in different regions which allows you to cut down on the number of requests as well as a couple other optimizations, and since video streams for 1080p (#43) and region-blocked videos need to be proxied through the instance anyway it also makes sense to use the same solution here.
Although I do very much agree that proxying has constraints I believe it is the best solution in this case.
perhaps an opt-in "proxy server" config so that my instance can proxy requests for other invidious instance? and pair that with a introduction system so servers can gossip about proxy servers they know, and which region are they.
@omarroth: I think I found a marker for geo-locked streams:
- get the direct URL to the video (e.g
https://r5---sn-q4fl6nly.googlevideo.com/videoplayback?expire=...). follow any redirects (redirector.googlevideo.com). - parse the query string; if it contains a parameter
gcr(i gotgcr=ruin my tests), it is geo/ip locked.
i only did limited testing; maybe you can try this out on your larger dataset.
In the future when reporting a video you should check which regions it's blocked in with this tool: http://polsy.org.uk/stuff/ytrestrict.cgi
For example this video is blocked worldwide except in Brunei: http://polsy.org.uk/stuff/ytrestrict.cgi?ytid=A-mjDxznYbE&auth=5ba28fe1d53f58118849ffc21bcda842
A 20$ bounty has been added to this issue.
Anyone opening a PR fixing this issue, will receive 20$ (in BTC) from the Invidious project.
More details: https://github.com/iv-org/invidious/issues/1898
We could use hola proxies which have a lot of available locations: https://github.com/Snawoot/hola-proxy
