cloudscraper icon indicating copy to clipboard operation
cloudscraper copied to clipboard
verified publisher icon VeNoMouS

cleanup Lib and CloudScraper TLS bypass is production ready

Open zinzied opened this issue 7 months ago • 9 comments

  • Final Clean Directory Structure
  • Performance Optimizations:
  • TLS Anti-Detection Features Confirmed Working

zinzied avatar Jun 10 '25 09:06 zinzied

@zinzied great work. Btw the system currently waits indefinitely for some cases.

For example trying to scrape a stack overflow url gets stuck. Running debug=True shows it's reached max concurrent request, even though there is currently only one request being processed.

masumsoft avatar Jun 10 '25 19:06 masumsoft

For example trying to scrape a stack overflow url gets stuck. Running debug=True shows it's reached max concurrent request, even though there is currently only one request being processed.

"Thanks for the additional context! Knowing you're using CloudScraper is really helpful, as it changes the potential source of the 'max concurrent requests' issue.

When CloudScraper gets stuck with that error, even on a single request, it often points to a specific challenge in how it's interacting with Cloudflare, or how its internal mechanisms are handling the anti-bot process. It's possible that a challenge is taking too long to resolve, or a session isn't being properly closed or released, leading CloudScraper to think it still has an 'active' (but stuck) request.

To help us pinpoint this, could you please provide a bit more detail?

  • CloudScraper Version: Which version of CloudScraper are you currently using?
  • Specific Code Snippet: Can you share the exact part of your code where you initialize and use CloudScraper for the URL? This helps us see how you're configuring it.
  • Full Debug Output: If possible, please paste the complete debug output when debug=True is enabled. CloudScraper's debug logs can often reveal detailed steps of the challenge-solving process, which might show exactly where it's hanging.

In the meantime, here are a few CloudScraper-specific troubleshooting steps you might try:

  • Explicit Session Closing: Ensure you're explicitly closing the CloudScraper session after you're done with it, especially if you're making multiple requests over time. For example: 
    import cloudscraper
scraper = cloudscraper.create_scraper()
try:
    response = scraper.get('your_url_here')
    # Process response
finally:
    scraper.close() # Ensure the session is closed
  • delay Parameter: Cloudflare can be sensitive to rapid requests. CloudScraper has a delay parameter (often used internally or configurable depending on the method) that might need adjustment if it's getting stuck waiting for a challenge.
  • User-Agent: Sometimes, using a different or more common User-Agent can help, though CloudScraper usually handles this.
  • Proxy Usage: Are you using any proxies with CloudScraper? If so, the proxy itself might be causing the hang-up.
  • Timeout Configuration: Confirm that the timeout parameter you're passing to CloudScraper is sufficient. Sometimes, solving Cloudflare challenges can take longer than a default timeout.
  • CloudScraper Updates: Make sure you're using the latest stable version of CloudScraper (pip install --upgrade cloudscraper). Updates often include fixes for new Cloudflare challenges or improved internal handling.

Once we have more information, especially the debug output, we'll be in a much better position to diagnose why CloudScraper is getting stuck and reporting max concurrent requests. Thanks for your patience!"

zinzied avatar Jun 11 '25 07:06 zinzied

@zinzied thanks for the quick reply. I'm directly using the code from your master branch. Here is my code snippet:

Installation

pip install git+https://github.com/zinzied/cloudscraper.git@8f13e9a9d1b1d8ff9108f713e3f9c8462cd37dce

Code Snippet

import cloudscraper
scraper = cloudscraper.create_scraper(browser={'custom': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36'}, debug=True)
response = scraper.get('https://stackoverflow.com/questions/381806/large-public-datasets', timeout=10)
print(response.status_code)

Debug Output

< GET /questions/381806/large-public-datasets HTTP/1.1
< Host: stackoverflow.com
< User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36
< Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
< Accept-Language: en-GB,en;q=0.9,en-US;q=0.8
< Accept-Encoding: gzip, deflate, br
< sec-ch-ua: "Google Chrome";v="117", "Not;A=Brand";v="8", "Chromium";v="117"
< sec-ch-ua-mobile: ?0
< sec-ch-ua-platform: "Windows"
< Sec-Fetch-Site: none
< Sec-Fetch-Mode: navigate
< Sec-Fetch-User: ?1
< Sec-Fetch-Dest: document
< DNT: 1
< 

> HTTP/1.1 403 Forbidden
> Date: Wed, 11 Jun 2025 13:08:15 GMT
> Content-Type: text/html; charset=UTF-8
> Transfer-Encoding: chunked
> Connection: close
> accept-ch: Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA
> cf-mitigated: challenge
> critical-ch: Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA
> cross-origin-embedder-policy: require-corp
> cross-origin-opener-policy: same-origin
> cross-origin-resource-policy: same-origin
> origin-agent-cluster: ?1
> permissions-policy: accelerometer=(),autoplay=(),browsing-topics=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
> referrer-policy: same-origin
> server-timing: chlray;desc="94e1560fafa4dae4"
> x-content-type-options: nosniff
> x-frame-options: SAMEORIGIN
> Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
> Expires: Thu, 01 Jan 1970 00:00:01 GMT
> Set-Cookie: __cf_bm=hpk.1lwPWC7KbW.yZQaiC1VSiPlnzGNuFTE.HSafFdc-1749647295-1.0.1.1-iFCeHxq9CNfEqTksNUQj.0AIIcVDOJoSG4GPYoiVo7xi3tI6kqmUdTO_KXwhqnXT.UD_1hwbk.6FwvdfhJFwAs1lEDNov81mxLjM2n.Jp_E; path=/; expires=Wed, 11-Jun-25 13:38:15 GMT; domain=.stackoverflow.com; HttpOnly; Secure; SameSite=None
> Vary: Accept-Encoding
> X-DNS-Prefetch-Control: off
> Server: cloudflare
> CF-RAY: 94e1560fafa4dae4-DAC
> Content-Encoding: gzip
> 
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=Edge"><meta name="robots" content="noindex,nofollow"><meta name="viewport" content="width=device-width,initial-scale=1"><style>*{box-sizing:border-box;margin:0;padding:0}html{line-height:1.15;-webkit-text-size-adjust:100%;color:#313131;font-family:system-ui,-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji}body{display:flex;flex-direction:column;height:100vh;min-height:100vh}.main-content{margin:8rem auto;max-width:60rem;padding-left:1.5rem}@media (width <= 720px){.main-content{margin-top:4rem}}.h2{font-size:1.5rem;font-weight:500;line-height:2.25rem}@media (width <= 720px){.h2{font-size:1.25rem;line-height:1.5rem}}#challenge-error-text{background-image:url(data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIzMiIgaGVpZ2h0PSIzMiIgZmlsbD0ibm9uZSI+PHBhdGggZmlsbD0iI0IyMEYwMyIgZD0iTTE2IDNhMTMgMTMgMCAxIDAgMTMgMTNBMTMuMDE1IDEzLjAxNSAwIDAgMCAxNiAzbTAgMjRhMTEgMTEgMCAxIDEgMTEtMTEgMTEuMDEgMTEuMDEgMCAwIDEtMTEgMTEiLz48cGF0aCBmaWxsPSIjQjIwRjAzIiBkPSJNMTcuMDM4IDE4LjYxNUgxNC44N0wxNC41NjMgOS41aDIuNzgzem0tMS4wODQgMS40MjdxLjY2IDAgMS4wNTcuMzg4LjQwNy4zODkuNDA3Ljk5NCAwIC41OTYtLjQwNy45ODQtLjM5Ny4zOS0xLjA1Ny4zODktLjY1IDAtMS4wNTYtLjM4OS0uMzk4LS4zODktLjM5OC0uOTg0IDAtLjU5Ny4zOTgtLjk4NS40MDYtLjM5NyAxLjA1Ni0uMzk3Ii8+PC9zdmc+);background-repeat:no-repeat;background-size:contain;padding-left:34px}@media (prefers-color-scheme:dark){body{background-color:#222;color:#d9d9d9}}</style><meta http-equiv="refresh" content="360"></head><body><div class="main-wrapper" role="main"><div class="main-content"><noscript><div class="h2"><span id="challenge-error-text">Enable JavaScript and cookies to continue</span></div></noscript></div></div><script>(function(){window._cf_chl_opt={cvId: '3',cZone: "stackoverflow.com",cType: 'managed',cRay: '94e1560fafa4dae4',cH: 'vxw.hlRnr1Y0BSveqZHSJODxXBRIVPb5eiWMYdDaxXI-1749647295-1.2.1.1-uoTJIbx_ChNkngoGHuc.e2OutY_5QRefIvjvYZswo_4n9EVenCGszTG9LccFHzHc',cUPMDTk: "\/questions\/381806\/large-public-datasets?__cf_chl_tk=ER_Ie7pY2Tv7XWN_vPN5XDEe4SInKeszbw2mmOJOD4k-1749647295-1.0.1.1-ye1gkNICJ4SJvV3CHnmUPHExeWqnirdKR0sE3z9a1v0",cFPWv: 'b',cITimeS: '1749647295',cTplC: 0,cTplV: 5,cTplB: 'cf',fa: "\/questions\/381806\/large-public-datasets?__cf_chl_f_tk=ER_Ie7pY2Tv7XWN_vPN5XDEe4SInKeszbw2mmOJOD4k-1749647295-1.0.1.1-ye1gkNICJ4SJvV3CHnmUPHExeWqnirdKR0sE3z9a1v0",md: "6.j6F1I00qcAVNyU0rfF1fketDVGwaQGE9mvsgezpgE-1749647295-1.2.1.1-x28fQ.64J8Qrk7qUodzcHKqSoexeLF0TCpIuzbdDB8lWLpPglFrbHIEiljhaW2poIHQMj649o5jQf292RJC5mAIh.pKTfyFzBX6LgapfTQ0dArHXz_COtRf4ZJXQetqVXAVaWwjEOSLQWSTKjw_qGkeAgB4Sio7ANQyirbgwXzt1NuJ4ZmVZjFiGxcwsXc5Ibj4Bk91Lid5C7Db0F3siCP.OqS6Oqr25X32PtbRjXB8b1NTlorbbaetYfjxf3wQCSQ0wHcDofqhCTw7qy2FwJnAwpjA2n.JGKl5nIoRxrQj06ZwOw_TqyTl3xSwtfrH9oy5Kso3kEbqY7M0q0rZMT4_wAyW2n._NTXyG0SqmS3D8NOYaBLUNacNu8_H9dYe2PtIdmOqi6i7SeS9WxSBAQ9aHQ406TeQtVLycHzT0NL.wBqXAsNxztzOgYFkkP752BluTERlvQ.nFvxesLD7xcOzPu5XiezJfocTKXf5Q4KZmJGvIIM2u_tjPROCf1tfFIT8L8AT7d4BVGjcpqroPWWvT_KUl.M8GveOGxYHx2hokxM2KdDQT1PvQnRZRvNybf.UlEqzm4EbW78a1485.6Zdklp1pn6AGkJ9jV79YSNP6rNjI35tMA4BUQzctNKIGbmzoulCZ3puqX6LCMtXLelPIQPPyw5eCzIy99G8Igylcr22BpOwgdRlGAOa5RvDAQSN8z_LGXwGZFg5zVvhpfjQt84rkj7RyQPe8Wmo1sDlwpt8ZbJ7XUsja2WYbayd0KhfuAGDTusAi2aP8AGVb_6LF9KqqkXhOY_GtGw5JJBcqm.ejlp9hNjaWqyH2Ucb1npkcHcVvSw3mdkC_TJT2lfEuvq6cUnj62qgoLCts0CG247CIBJ8kYGOX0yiyWyL5xVEqmpvsCp7bQ7t9_6WzvLR.u7ht5ysC5VU5D_PIEzchmoT8GAv1yEjHUS4ETrFRUTDnKxODi_baDgB3XIqFjsLthgRcb.pnMV5s3Y0WE.1ClxapqSJw2uh2VqockECTcACyskfAQNsVw4lpWRMHPVsWkuCArOS7A0M3aKPAO6uhKYBMu12EAH.stUSGrV9L",mdrd: "1FwbxugT7MNedqD9YQ0IjosYQpWgU2HSa5THp_YvvXc-1749647295-1.2.1.1-lF9dWKhC8eI7q43LgkBe8RWQNQyCCySGRWO07XN7lXosrtPL6Zg77NtPzSHu3AhoTYTzdZfJuQPo5qHMhKIfNmLPzr1ZX3A7NhWdeJQNoR7AVuTl6dt__nJ9mOK0R5m5ZhbP5j.aUwjOJ_ATKQZkl5EaSe75n6zjPqP3Aa6btUMmuJSX5oy5vPSouA84loJS6C0bSjUeRsy9uZMdAi51oaLMcG7S3LOYErKlJG7ckJ_LUxMCi6thLXGVhqYmRnhApE4bej1TB8SvCXfeQyPQn1vT1oiP4eNT3CU.2D7wx73s6TXGBWGimThxXGTlg5pVU5mV6TwTO_QN6WVSKaTLLKa5fkuYiIYUjLJR.SPUBcp_0PZp2ezzwNW1WPJ8JM.BOQxFlzGGMMBksHf_j8aZqFtgT9r_kzQzc9J_1uO2k17R9aYjGBp69SJwQZ43poYaAQoOdh8pM78SF2BCfS4akgSgZA89ci_xemrwMaHFoSsMkz0th2AZ5b8a12U9MsPuIc7dYm6GGw8H.kt3VRpw6THT3WWuM.p2pPkDQr_mzhYEXpTxO50Q1CV9PAGn8IM2hUW5e0k_7bcy_tdyoEby2dAeuqsSKdhH3hJc_d54HeusPnCzs.xchyHPAoqWaWbw4Sii1awiVuKfc2MpWJq.kUeynMaF5HFGhlVb1D4cEEzqmS3z.r0Og1XOsgHowezFwFPN0b4j6fKQTVKE3AKcTIhab7awWjqkvq9XcemtITAfxck3TK9yul87L2ZIvt.H5dtl13WGP_vz.XY7JUV4ptM5vPuYLJCaqAFYV6uLWxy.lBvFGzc.tOdNGrLP1.FRHyIhXM6_w_G9AIdKgg3dlywITVp4c_XLICs1KcE5hNvzHeK3AIeYvCRGdH1OkIegGvME_J_GO1zghYTZVbLsjKh0adH6UDefpONjvOrH_1dpN82ZiWFY8i05LMSYpk9dFQgSW6vfV9m4Lk81Y0.rAFLp6j5F13BxGEg3vfiMf14i_qLtAoNGQ_lL8ZjMmofzAdTTYHCXvD1uBF3NjfUGFVavCCseUHeJfb2q7aXnyuTGFlUZwKR8VneH0t4qSH2fTFrwnFZmMqqw8sjS5OMh9XEl7tAQNAWp.eIaJixGIBcIx5nQDQnPfGwEKclRD6vSyZwo5Bolyz.E2c2q4y6KxAImtCr11AXPHxPh7GUOfmZ3nMYE6TAaLJMizzEmIX3CLURHjsei0NzKnWWa2leGG5VU2GDLMuJESrMQ5Zj.3Ilqd0vB0ABhwgOq0yRgIPFOjH.eoJ3cNfeL85UJqRC1dZJpZbATON66KwhSYqvCuv8CTOU9zMRF2p2B9dvJgSDGFLmCdUyBF4oXtDpOaK_VzLcL9.HS7YfpAYnALOIjjHGYwY.JaenG8ksDcxQRoenpo3Fjpswr0y5Q9draIWpYtZJxiBCuBWIKARLKIti1vDJWDL3x8gLR0OP3zNpqOns7IFLctBwUsrwmvJD4tftns0Y55JMlN8GaSCTpBtj021eH_SlHEnXNFpzeGGT8hcAkPOgxHY5qzQG8nNV0Q5Grac5Jgs3b0eLCGucPoAojA29Zzt2RkGWNdEYCjtEcK0lmkrzc70sPeyp879VZnlDK4nSUMkQLHk3kqThtE9u.maJEzwE0yRVnCem.m9wAoB_T1FkLqUGNK664Z7xZ_mAmmo..iqG9TyAUrbN_zUvreXOaIc6BeGyAg67srSc5jSdGLP06g1w2WMjCkYXS4pKcZhy81F7V0WFLLkzVdT20P3gMYIXMmaOKlgIUuFQ1j.vzblGWcIAQdY84roWCzoDCqfEOJJGhgwrPHg8z7SygZIUyFTQy9ARiGzHkUjRbuqEBJBbtzh9xUn4QyAVz6tG2D16gx655pXNeMx9qUcwxK5R.hJ6LTPhy9XvNkPunrQ27uZMYi_dOhBCQiWu_yh1pAWiVLgK8glbvvRFGrMliA.q7pFLYw6S_noGbEXb9cK3MaMt5GnhjMS3tmCEsWm3zrgdC4_gI4ci2i1ZU0QpGcqA8t1w6ZMniSCcU4pwzIzF9WYrGRRmECqi5fQWpho_4g1.qyspTjak4sbG69t0QGeq2StnOhxL570WPryILRwivSiq6P3q3JF.F3Nc7ycbujtbb9LuYAWmepS2kWLxwJvaHjksF4synoVp.LqG3P_Z4WDzv.nL7GhSxoQriyEEf2qxZw2CfrZNyVHtFO.8AtUlnchXavr6xcWV5Rj0QYgOKP..yFIronUM59pUQ_Pt9jbTtmXqJWuJqx2hvtdLd1nOHCDYA5IuuNpsh51mnIT9.gZT26RpMSpR_qbL6mIsrP8AnhfVgmNQVAzohcD7ORJ2hUbIOdP6T5PD4iPCQxP_IwrszyLrpa0xbo83idyeB2dDIDu7oixiP1kH_.Vdf.bRIVroaBnxoHdRDy0lvM87K2QHBRIlbXG.kvCBSM_jdiocMTCvQvrMjFl.S9JqPnIZ9BIHTMyxEgt_aPyCSWNp9d8oxHCqS7A1cwaiJNV1NwewKXZWaXMA5e_i.ksTgWT1t4Q.4PD7Gwh8lGdUVPlLHunRJ6Vi33NiEgu4zWBFgdoHEjUjGBNjT2qIH2l_2hvi6qXvnCgGOHiqpBfF6xPOgwxWdGyCXAhEGYEqplwzvoeFBcHw.QMJ_ARLLpJTZ57a6cKQ09PyI4kFAyK.VHsEd",};var a = document.createElement('script');a.src = '/cdn-cgi/challenge-platform/h/b/orchestrate/chl_page/v1?ray=94e1560fafa4dae4';window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;if (window.history && window.history.replaceState) {var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;history.replaceState(null, null, "\/questions\/381806\/large-public-datasets?__cf_chl_rt_tk=ER_Ie7pY2Tv7XWN_vPN5XDEe4SInKeszbw2mmOJOD4k-1749647295-1.0.1.1-ye1gkNICJ4SJvV3CHnmUPHExeWqnirdKR0sE3z9a1v0" + window._cf_chl_opt.cOgUHash);a.onload = function() {history.replaceState(null, null, ogU);}}document.getElementsByTagName('head')[0].appendChild(a);}());</script></body></html>
Refreshing session due to staleness or 403 errors...
Cleared Cloudflare cookies for session refresh
⏱️ Request throttling: sleeping 0.95s
🚦 Concurrent request limit reached (1/1), waiting...

masumsoft avatar Jun 11 '25 13:06 masumsoft

Note that after trying your latest commit (2b909127bef6850d263e8c99ad00c4114fa6f322) on the same code this is now an infinite loop:

Debug Output

Refreshing session due to staleness or 403 errors...
Cleared Cloudflare cookies for session refresh
🔄 Temporarily reset concurrent counter for session refresh (was 1)
⏱️ Request throttling: sleeping 0.80s
🔐 Rotated TLS cipher suite (rotation #1)
    Using 8 ciphers starting from index 1
Refreshing session due to staleness or 403 errors...
Cleared Cloudflare cookies for session refresh
🔄 Temporarily reset concurrent counter for session refresh (was 0)
⏱️ Request throttling: sleeping 0.98s
🔐 Rotated TLS cipher suite (rotation #2)
    Using 8 ciphers starting from index 2
Refreshing session due to staleness or 403 errors...
Cleared Cloudflare cookies for session refresh
🔄 Temporarily reset concurrent counter for session refresh (was 0)
⏱️ Request throttling: sleeping 0.97s
🔐 Rotated TLS cipher suite (rotation #3)
    Using 8 ciphers starting from index 3
Refreshing session due to staleness or 403 errors...
Cleared Cloudflare cookies for session refresh
🔄 Temporarily reset concurrent counter for session refresh (was 0)
⏱️ Request throttling: sleeping 0.97s
🔐 Rotated TLS cipher suite (rotation #4)
    Using 8 ciphers starting from index 4
Refreshing session due to staleness or 403 errors...
Cleared Cloudflare cookies for session refresh
🔄 Temporarily reset concurrent counter for session refresh (was 0)
⏱️ Request throttling: sleeping 0.97s
🔐 Rotated TLS cipher suite (rotation #5)
    Using 8 ciphers starting from index 5
Refreshing session due to staleness or 403 errors...
Cleared Cloudflare cookies for session refresh
🔄 Temporarily reset concurrent counter for session refresh (was 0)
⏱️ Request throttling: sleeping 0.98s
🔐 Rotated TLS cipher suite (rotation #6)
    Using 8 ciphers starting from index 6
Refreshing session due to staleness or 403 errors...
Cleared Cloudflare cookies for session refresh
🔄 Temporarily reset concurrent counter for session refresh (was 0)
⏱️ Request throttling: sleeping 0.98s
🔐 Rotated TLS cipher suite (rotation #7)
    Using 8 ciphers starting from index 7
Refreshing session due to staleness or 403 errors...
Cleared Cloudflare cookies for session refresh
🔄 Temporarily reset concurrent counter for session refresh (was 0)
⏱️ Request throttling: sleeping 0.98s
🔐 Rotated TLS cipher suite (rotation #8)
    Using 8 ciphers starting from index 8
Refreshing session due to staleness or 403 errors...
Cleared Cloudflare cookies for session refresh
🔄 Temporarily reset concurrent counter for session refresh (was 0)
⏱️ Request throttling: sleeping 0.98s
🔐 Rotated TLS cipher suite (rotation #9)
    Using 8 ciphers starting from index 9
Refreshing session due to staleness or 403 errors...
Cleared Cloudflare cookies for session refresh
🔄 Temporarily reset concurrent counter for session refresh (was 0)

masumsoft avatar Jun 11 '25 13:06 masumsoft

I used proxy, TLS rotation and sticky proxy IP information. But after running it for a while, it couldn't bypass Cloudflare. I think it was blocked by Cloudflare. Maybe it was because the request TLS was blocked, or maybe it was something else. I paused it for an hour or two, and then ran it again, and it was able to bypass Cloudflare normally. Can you fix this?

 while True:
    try:
        device_id = str(uuid.uuid4().int)[:10]  # 截取10位数字
        state = str(uuid.uuid4())
        url = f'https://accounts.moneylion.com/authorize?clientId=a611942d-d22e-4c24-8f8e-60f08b01cbfc&redirectUri=https%3A%2F%2Fweb.moneylion.com%2Fauth%2Fcallback&deviceId={device_id}&flowEntry=signin&scope=profile&responseType=code&state={state}'
        proxies = [
            'http://user:[email protected]:6001'.replace('randonm', generate_random_10())
        ]
        ua = get_ua()
        headers = {
            'User-Agent': ua
        }
        scraper = cloudscraper.create_scraper(
            rotating_proxies=proxies,
            delay=5,  # Longer delay for complex challenges
            interpreter='js2py',  # Most compatible interpreter
            enable_stealth=True , # Additional stealth for v3 detection
            auto_refresh_on_403 = False,
            max_403_retries = 8,
            debug = True,
        )
        response = scraper.get(url,headers = headers)
        if response.status_code == 200 or response.status_code == 302:
            logger.info('cookie获取成功')
            return scraper
        scraper.close()
    except:
        pass

3279444869 avatar Jun 24 '25 15:06 3279444869

@zinzied - Take a look at https://github.com/VeNoMouS/cloudscraper/issues/298 (TLS Fingerprint Support). I wasn't able to solve this PR, but perhaps you have some insights based on your work.

rsforbes avatar Jun 29 '25 15:06 rsforbes

@zinzied - Take a look at #298 (TLS Fingerprint Support). I wasn't able to solve this PR, but perhaps you have some insights based on your work.

CloudScraper V3 Handler Enhancements

What This Project Accomplished

I set out to solve a specific problem: accessing prosportstransactions.com, which was blocked by Cloudflare's advanced protection. While I didn't achieve the original goal due to fundamental technical limitations, I made significant improvements to CloudScraper that benefit the entire community.

The Challenge

Modern websites like prosportstransactions.com use sophisticated protection mechanisms that operate at multiple layers:

  1. Application Layer: JavaScript challenges that need to be solved
  2. Transport Layer: TLS fingerprint detection that identifies automated tools

What I Improved ✅

Enhanced Challenge Detection

I think ill upgrade CloudScraper's ability to recognize modern Cloudflare challenges by adding support for:

  • window._cf_chl_opt structures: New challenge format used by modern sites
  • Improved pattern matching: Better regex patterns that catch more challenge types
  • Robust error handling: Graceful fallbacks when parsing fails

zinzied avatar Jul 09 '25 16:07 zinzied

Unlike #295 this is not resolving the loopig issue on the TLS

vincentkoc avatar Aug 30 '25 17:08 vincentkoc

I tried to use this fork, as I like the robustness it offers, but there was an issue with recursion in the CipherSuiteAdapter in your init.py. I was able to get it working by monkey patching the VeNoMous CipherSuiteAdapter in your fork and it works as expected without recursion.

tlg7c5 avatar Sep 23 '25 17:09 tlg7c5