kenpompy
kenpompy copied to clipboard
j-andrews7
login failure
Hello,
I am getting the following error when trying to use the login function: LinkNotFoundError()
I am using the following code:
browser = login(email, password)
This used to work fine last season
Appreciate the help!
Here is the response im getting. Seems it is being blocked by Cloudflare...
Sorry, you have been blocked
You are unable to access kenpom.com
<span class="cf-no-screenshot error"></span>
</div>
</div>
</div><!-- /.captcha-container -->
<div class="cf-section cf-wrapper">
<div class="cf-columns two">
<div class="cf-column">
<h2 data-translate="blocked_why_headline">Why have I been blocked?</h2>
<p data-translate="blocked_why_detail">This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.</p>
</div>
<div class="cf-column">
<h2 data-translate="blocked_resolve_headline">What can I do to resolve this?</h2>
<p data-translate="blocked_resolve_detail">You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.</p>
</div>
</div>
</div><!-- /.section -->
Thanks for the report! This was previously discovered in #24. A patch was subsequently issued and merged into master as part of #25, however the release on PyPi hasn't caught up to what we've got here.
If you need this functionality in the immediate term, you could install from the GitHub source but I imagine @j-andrews7 will push a new release with all the bug fixes we've got in the hopper before the season starts.
Yeah, I'll push a release to PyPi. And probably try to get stuff set up so Sean can push them too. Real work is making it really hard to carve out any time for this lately.
On Thu, Oct 27, 2022, 10:37 AM Sean Quinn @.***> wrote:
Thanks for the report! This was previously discovered in #24 https://github.com/j-andrews7/kenpompy/issues/24. A patch was subsequently issued and merged into master as part of #25 https://github.com/j-andrews7/kenpompy/pull/25, however the release on PyPi hasn't caught up to what we've got here.
If you need this functionality in the immediate term, you can install from the GitHub source but I imagine @j-andrews7 https://github.com/j-andrews7 will push a new release with all the bug fixes we've got in the hopper before the season starts.
— Reply to this email directly, view it on GitHub https://github.com/j-andrews7/kenpompy/issues/33#issuecomment-1293719357, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOAQNH4ZAPYMH46PVZDKCDWFKOUNANCNFSM6AAAAAARO3A3DQ . You are receiving this because you were mentioned.Message ID: @.***>
Will try to push a new release tonight.
On Mon, Nov 7, 2022, 8:11 AM trludt @.***> wrote:
Any update on this issue?
— Reply to this email directly, view it on GitHub https://github.com/j-andrews7/kenpompy/issues/33#issuecomment-1305674731, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOAQND3LDPPTANUVW6UF5TWHEEZDANCNFSM6AAAAAARO3A3DQ . You are receiving this because you were mentioned.Message ID: @.***>
This should be resolved in the latest release. Or at least the tests pass. If everything's still broken, blame @esqew.
Do not actually do that - he's the only reason this ever got fixed. Thanks Sean!
pip install --upgrade kenpompy should get you the latest version with this fix and a few others.
Hm. @esqew, have you noticed any inconsistencies with this? Working fine with python 3.8-3.10.
@trludt can you try a few more times and see if you can get through? We are somewhat at Ken's power here, and if for whatever reason your IP is subject to more stringent measures, there's likely not a ton we can do.
I've tried a bunch still no luck... This is wild. I probably pulled data once per day of the season last year. Could that have restricted my IP? And does a VPN change anything?
I asked Ken about this before I published it, and he basically said as long as people didn't abuse it, he was totally cool with it. I can't imagine once a day is a problem.
Not sure about the VPN, though it could make sense if it's changing your IP geolocation to an area identified as more likely to be an attack.
I just shot him an email. Hopefully he gets back to me. I've tried the browser = (email, pass) line at least 50 times today and still errors every time.
FYI I'm using Spyder v5.1.5 and Python 3.9.12 in my script if that helps towards anything
I unfortunately cannot reproduce this on my end, and the CI/CD tests checked out fine last night.
@trludt Where does your VPN reside geographically? Cloudflare will more closely scrutinize connections from IPs in higher-risk areas. Separately, can you provide a stripped down example that demonstrates this? Is it possible you're also trying to log in a bunch of times or scrape a ton of data in a short period?
I am always running into the Cloudflare error on that login step. I'm not always using the VPN (ExpressVPN) but when I am using it, my connection runs through primarily big US cities (Atlanta, Dallas, Phoenix)
Thanks for the context. Without being able to reproduce this issue, it's likely this is something relating to how Cloudflare is handling your specific connection or sessions.
It's possible Cloudflare's heuristics have detected either/both that (among other possibilities) (a) your account has logged in from multiple geographies in too short of a time span ("impossible travel" anomaly detection), or (b) your script, at one point or another, launched too many requests and Cloudflare is now taking it upon itself to block further requests for an indefinite amount of time.
If you're able, can you set your debugger to break on this exception and dump the contents of str(browser.page.contents) here to ensure that this exception is firing correctly at the very least?
Would that be in my script or altering the utils.py file under the Kenpompy code? @esqew
Neither, but there may be an easier way now that I think about it, this simplified script may be easier to get the contents of the page that Cloudflare is throwing to see if there's anything specific about your case we can pick out from it:
from kenpompy.utils import login
try:
browser = login('<email>', '<password'>)
except Exception as e:
print(str(browser.page.contents))
raise e
Ultimately what I would really like to understand is what HTTP status code Cloudflare is throwing to see if that might be more illuminating in this situation, but the way we've got MechanicalSoup set up doesn't currently capture the right details to enable this. Hopefully the HTML Cloudflare is sending back will indicate something helpful to preclude this next level of debugging.
`['html', '[if lt IE 7]> <![endif]', '[if IE 7]> <![endif]', '[if IE 8]> <![endif]', '[if gt IE 8]><!',
Sorry, you have been blocked
You are unable to access kenpom.com
Why have I been blocked?
This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.
What can I do to resolve this?
You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.
['html', '[if lt IE 7]> <![endif]', '[if IE 7]> <![endif]', '[if IE 8]> <![endif]', '[if gt IE 8]><!',
Sorry, you have been blocked
You are unable to access kenpom.com
Why have I been blocked?
This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.
What can I do to resolve this?
You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.
So it's definitely a Cloudflare interstitial (so the exception itself is working as intended). Unfortunately without a reliable way to reproduce this behavior outside your specific environment there's not much else we can advise you to do. Cloudflare deliberately doesn't share a ton of details to make it more difficult to circumvent their protections. Unless you're able to get in touch with KenPom himself (or someone who handles his Cloudflare account) and can look at the logs specific to the Cloudflare Ray ID values from the markup you posted, we're just shooting in the dark.
If you have the resources available, you may consider attempting to try this same code from another machine and/or using another KenPom account to see if it's reproducible across environments/accounts. The only other course of action I could recommend at this point would be to drop this altogether for at least a couple days to let any rate limiting that may be at play get fully reset.
I tried a different KenPom account and that didn't work. Also tried from my PC instead of my Macbook, no luck. I shot Ken an email with the Cloudflare Ray ID and my IP... hopefully I can get unbanned 🙏
This tool rocks and would hate to lose access to it. Truly had no intention of abusing it or overloading his site.
I appreciate @esqew @j-andrews7 for the assistance, much appreciated!
@trludt Please let me know if he reaches out to you on this issue. I am having similar results being blocked by Cloudflare. I originally had my own scraping pipeline developed through beautifulsoup that I used last year (about daily similar to you). Recently I ran my script to see if it still worked for this season and I received the 403 forbidden code. I then tried this python library as an alternative and received the same block. In my personal pipeline I also tried modifying User-Agent status and several other request headers and nothing resolved the issue.
@trludt and @AVA-27 I have also received the same LinkNotFoundError (now is replaced by Exception: Opening kenpom.com failed - request was intercepted by Cloudflare protection) after forcing kenpompy update via pip)
I have also reached out to Ken via email and will let you know if I hear anything.
For context, I use a VPN (NordVPN) out of Atlanta (physically located in TN). Last night at about 2:00 AM CST I received the first error (on VPN), and tried again disconnected (still received LinkNotFound error).
Today, after updating, I explicitly tried to re-run while being disconnected from VPN to see if there was a difference, but I'm afraid I might already be blacklisted b/c of the re-geotagging of my first request via VPN.
@RobMepham I never utilized a VPN to access the website and still cannot gain access so I am not sure if that is playing a role or not. Edit: I also reached out to Ken via email; if I receive a response I will update as well.
Good to know I'm not alone! Currently working on migrating my script over to a lambda function on AWS, which was ultimately my plan all along. Maybe this is a good thing my hand is being forced 🤣