oembetter
oembetter copied to clipboard
apostrophecms
Too many youtube requests leads to bot suspicion.
Story: We have a decent amount of users visiting our website with youtube embedded videos on some pages. Suddenly all our videos broke up. We have tried to reproduce it in a development(or local) environment - unsuccessfully. After long production debugging we have found the source of our struggles: This request to youtube: https://github.com/apostrophecms/oembetter/blob/5d728fa2b930807719fc66c8aa43d7b4d3629f36/oembed.js#L33 started to return unexpected content:
<!DOCTYPE html><html><head><title>YouTube</title><script src="https://www.google.com/recaptcha/api.js?hl=en" async defer nonce="O62rLnO7QtHphxso6PaSoA"></script><style nonce="O62rLnO7QtHphxso6PaSoA">html{height:100%;margin:0;padding:0;border:0;font-size:100%;background:transparent}body{margin:0;padding:0;border:0;font:12px "YouTube Noto",Roboto,arial,sans-serif;background:#f1f1f1;height:100%}#captcha-page{text-align:center;height:100%;white-space:nowrap;word-break:normal}.g-recaptcha{display:inline-block}#captcha-page-content{vertical-align:middle;display:inline-block}#captcha-page-vertical-align{height:100%;display:inline-block;vertical-align:middle}p{margin:auto;color:#333;font-size:16px;white-space:normal;text-shadow:0 0 0 transparent,0 1px 1px #fff}</style><link rel="shortcut icon" href="https://www.youtube.com/img/favicon.ico" type="image/x-icon"><link rel="icon" href="https://www.youtube.com/img/favicon_32.png" sizes="32x32"><link rel="icon" href="https://www.youtube.com/img/favicon_48.png" sizes="48x48"><link rel="icon" href="https://www.youtube.com/img/favicon_96.png" sizes="96x96"><link rel="icon" href="https://www.youtube.com/img/favicon_144.png" sizes="144x144"></head><body><div id="captcha-page"><div id="captcha-page-content"><p>Sorry for the interruption. We have been receiving a large volume of requests from your network.</p><p>To continue with your YouTube experience, please fill out the form below.</p><form action="/das_captcha?fw=1" method="POST"><div class="g-recaptcha" data-sitekey="6Lf39AMTAAAAALPbLZdcrWDa8Ygmgk_fmGmrlRog"></div><br/><input type="hidden" name="action_recaptcha_verify2" value="1"><input type="hidden" name="next" value="/watch?v=Z9JWY_8JRKA&t=2s"><input type="submit" value="Submit"></form><style nonce="O62rLnO7QtHphxso6PaSoA">#yt-masthead{margin:15px auto;width:440px;margin-top:25px}#logo-container{margin-right:5px;float:left;cursor:pointer;text-decoration:none}.logo{background:no-repeat url("//www.gstatic.com/youtube/img/branding/youtubelogo/1x/youtubelogo_30.png");width:125px;height:30px;cursor:pointer;display:inline-block}#masthead-search{display:flex;margin-top:3px;max-width:650px;overflow:hidden;padding:0;position:relative}.search-button{border-left:0;-moz-border-radius-topleft:0;border-top-left-radius:0;-moz-border-radius-bottomleft:0;border-bottom-left-radius:0;float:right;height:29px;padding:0;border:solid 1px transparent;border-color:#d3d3d3;background:#f8f8f8;color:#333;cursor:pointer}.search-button:hover{border-color:#c6c6c6;background:#f0f0f0;box-shadow:0 1px 0 rgba(0,0,0,0.10)}.search-button-content{border:none;display:block;opacity:.6;padding:0;text-indent:-10000px;background:no-repeat url(//www.gstatic.com/youtube/src/web/htdocs/img/search.png);background-size:auto;width:15px;height:15px;box-shadow:none;margin:0 25px}#masthead-search-terms-border{flex:1 1 auto;border:1px solid #ccc;box-shadow:inset 0 1px 2px #eee;background-color:#fff;font-size:14px;height:29px;line-height:30px;margin:0 0 2px;overflow:hidden;position:relative;-moz-box-sizing:border-box;box-sizing:border-box;-webkit-transition:border-color .2s ease;transition:border-color .2s ease}#masthead-search-terms{background:transparent;border:0;font-size:16px;height:100%;left:0;margin:0;outline:none;padding:2px 6px;position:absolute;width:100%;-moz-box-sizing:border-box;box-sizing:border-box}</style><div id="yt-masthead"><a id="logo-container" href="https://www.youtube.com/" title="YouTube home"><span class="logo" title="YouTube home"></span></a><form id="masthead-search" class="search-form" action="https://www.youtube.com/results" onsubmit="if (document.getElementById('masthead-search-terms').value == '') return false;"><div id="masthead-search-terms-border" dir="ltr"><input id="masthead-search-terms" autocomplete="off" onkeydown="if (!this.value && (event.keyCode == 40 || event.keyCode == 32 || event.keyCode == 34)) {this.onkeydown = null; this.blur();}" name="search_query" value="" type="text" placeholder="Search" title="Search" aria-label="Search"></div><button class="search-button" type="submit" onclick="if (document.getElementById('masthead-search-terms').value == '') return false; document.getElementById('masthead-search').submit(); return false;" dir="ltr"><span class="search-button-content">Search</span></button></form></div></div><span id="captcha-page-vertical-align"></span></div></body></html>
Sorry for the interruption. We have been receiving a large volume of requests from your network. To continue with your YouTube experience, please fill out the form below.
Basically what happened, is that youtube suspected our production IP-address at being a bot :D
After all we have ended up with a self made embedded video widget... but wanted to inform you that this problem exists.
Thanks for passing that on! We have worked really hard to keep making true oembed queries to youtube, and we do cache the results rather than hitting them every time, so it is a little surprising that this happens.
If you have any data on the requests that were being made we'd be interested to see it. Perhaps we're not caching failures enough and that's leading to too many retries on bad video links and getting you banned. Or perhaps you just have so many videos that even the once an hour requests to refresh the cache are too many.
Well, there is one video on average on each page. One page has 10 videos though >< Also, we are running 2-6 node containers at same time. I guess it was only a matter of time...
Links were not broken. I did not find anything weird in logs.. Youtube just started sending this response, that basically it.
OK, thanks. Maybe we're not caching errors or something. Otherwise I don't see how we'd hit a problematic number of requests. Very strange.
On Mon, Feb 8, 2021 at 10:24 AM fedoseiev [email protected] wrote:
Well, there is one video on average on each page. One page has 10 videos though >< Also, we are running 2-6 node containers at same time. I guess it was only a matter of time...
Links were not broken. I did not find anything weird in logs.. Youtube just started sending this response, that basically it.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/apostrophecms/oembetter/issues/12#issuecomment-775225583, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAH27JWCAZMF52MXB6A3GLS576Z7ANCNFSM4XGEWT5A .
--
THOMAS BOUTELL | CHIEF TECHNOLOGY OFFICER APOSTROPHECMS | apostrophecms.com | he/him/his
(The cache is shared across processes.)
On Mon, Feb 8, 2021 at 11:18 AM Tom Boutell [email protected] wrote:
OK, thanks. Maybe we're not caching errors or something. Otherwise I don't see how we'd hit a problematic number of requests. Very strange.
On Mon, Feb 8, 2021 at 10:24 AM fedoseiev [email protected] wrote:
Well, there is one video on average on each page. One page has 10 videos though >< Also, we are running 2-6 node containers at same time. I guess it was only a matter of time...
Links were not broken. I did not find anything weird in logs.. Youtube just started sending this response, that basically it.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/apostrophecms/oembetter/issues/12#issuecomment-775225583, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAH27JWCAZMF52MXB6A3GLS576Z7ANCNFSM4XGEWT5A .
--
THOMAS BOUTELL | CHIEF TECHNOLOGY OFFICER APOSTROPHECMS | apostrophecms.com | he/him/his
--
THOMAS BOUTELL | CHIEF TECHNOLOGY OFFICER APOSTROPHECMS | apostrophecms.com | he/him/his