cloudscraper icon indicating copy to clipboard operation
cloudscraper copied to clipboard

Not Working

Open darkman13 opened this issue 2 months ago • 11 comments

Summary.

Expected Result

To get the html content but the scraper is now 403 blocked

Actual Result

403 - permuting over many possible scraper configs still didn't work :/

import cloudscraper

scraper = cloudscraper.create_scraper(debug = True)  # returns a CloudScraper instance
print(cloudscraper.__version__)

print(scraper.get("https://www.masterofmalt.com/whiskies/springbank/hazelburn-sherry-8-year-old-whisky").status_code)

result

1.2.71
< GET /whiskies/springbank/hazelburn-sherry-8-year-old-whisky HTTP/1.1
< Host: www.masterofmalt.com
< User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
< Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
< Accept-Language: en-US,en;q=0.9
< Accept-Encoding: gzip, deflate
<

> HTTP/1.1 403 Forbidden
> Cache-Control: private, no-store, max-age=0
> Content-Type: text/html; charset=utf-8
> Server: Vercel
> X-Vercel-Challenge-Token: 2.1761144392.60.NzlkOGRmYjBmYmRkMzNlMDBkOTI3MGE3NmI1NmQyZDE7Nzc0ZDU2MTg7ZjcyMWYxMWM0NmIwNzc5N2NhZTlmNDZkYTBjMDZiM2Q3NDc4NGYxZTszOxNB7NEUYLTCNca4XpTIzJDEauEGizJTluLcxv5IZBtoFl1Q2wk4zLP6aOHKOtA0TVc=.38a8f31efa52a606ada17f334048a98f
> X-Vercel-Id: icn1::1761144392-9m9ExmVWciMffM1mcPy5twm3Q8vGpHHP
> X-Vercel-Mitigated: challenge
> Date: Wed, 22 Oct 2025 14:46:32 GMT
> Transfer-Encoding: chunked
>
<!DOCTYPE html><html lang="en" data-astro-cid-nbv56vs3> <head><meta charset="utf-8"><meta 
........
</div> </noscript>   </body></html>

Reproduction Steps

System Information

$ python -m cloudscraper.help
{
    "platform": {
        "system": "Darwin",
        "release": "24.6.0"
    },
    "interpreter": {
        "name": "CPython",
        "version": "3.13.3"
    },
    "cloudscraper": "1.2.71",
    "requests": "2.32.4",
    "urllib3": "2.5.0",
    "OpenSSL": {
        "version": "OpenSSL 3.5.1 1 Jul 2025",
        "ciphers": [
            "ADH-AES128-GCM-SHA256",
            "ADH-AES128-SHA",
            "ADH-AES128-SHA256",
            "ADH-AES256-GCM-SHA384",
            "ADH-AES256-SHA",
            "ADH-AES256-SHA256",
            "ADH-CAMELLIA128-SHA",
            "ADH-CAMELLIA128-SHA256",
            "ADH-CAMELLIA256-SHA",
            "ADH-CAMELLIA256-SHA256",
            "AECDH-AES128-SHA",
            "AECDH-AES256-SHA",
            "AES128-CCM",
            "AES128-CCM8",
            "AES128-GCM-SHA256",
            "AES128-SHA",
            "AES128-SHA256",
            "AES256-CCM",
            "AES256-CCM8",
            "AES256-GCM-SHA384",
            "AES256-SHA",
            "AES256-SHA256",
            "ARIA128-GCM-SHA256",
            "ARIA256-GCM-SHA384",
            "CAMELLIA128-SHA",
            "CAMELLIA128-SHA256",
            "CAMELLIA256-SHA",
            "CAMELLIA256-SHA256",
            "DHE-DSS-AES128-GCM-SHA256",
            "DHE-DSS-AES128-SHA",
            "DHE-DSS-AES128-SHA256",
            "DHE-DSS-AES256-GCM-SHA384",
            "DHE-DSS-AES256-SHA",
            "DHE-DSS-AES256-SHA256",
            "DHE-DSS-ARIA128-GCM-SHA256",
            "DHE-DSS-ARIA256-GCM-SHA384",
            "DHE-DSS-CAMELLIA128-SHA",
            "DHE-DSS-CAMELLIA128-SHA256",
            "DHE-DSS-CAMELLIA256-SHA",
            "DHE-DSS-CAMELLIA256-SHA256",
            "DHE-PSK-AES128-CBC-SHA",
            "DHE-PSK-AES128-CBC-SHA256",
            "DHE-PSK-AES128-CCM",
            "DHE-PSK-AES128-CCM8",
            "DHE-PSK-AES128-GCM-SHA256",
            "DHE-PSK-AES256-CBC-SHA",
            "DHE-PSK-AES256-CBC-SHA384",
            "DHE-PSK-AES256-CCM",
            "DHE-PSK-AES256-CCM8",
            "DHE-PSK-AES256-GCM-SHA384",
            "DHE-PSK-ARIA128-GCM-SHA256",
            "DHE-PSK-ARIA256-GCM-SHA384",
            "DHE-PSK-CAMELLIA128-SHA256",
            "DHE-PSK-CAMELLIA256-SHA384",
            "DHE-PSK-CHACHA20-POLY1305",
            "DHE-RSA-AES128-CCM",
            "DHE-RSA-AES128-CCM8",
            "DHE-RSA-AES128-GCM-SHA256",
            "DHE-RSA-AES128-SHA",
            "DHE-RSA-AES128-SHA256",
            "DHE-RSA-AES256-CCM",
            "DHE-RSA-AES256-CCM8",
            "DHE-RSA-AES256-GCM-SHA384",
            "DHE-RSA-AES256-SHA",
            "DHE-RSA-AES256-SHA256",
            "DHE-RSA-ARIA128-GCM-SHA256",
            "DHE-RSA-ARIA256-GCM-SHA384",
            "DHE-RSA-CAMELLIA128-SHA",
            "DHE-RSA-CAMELLIA128-SHA256",
            "DHE-RSA-CAMELLIA256-SHA",
            "DHE-RSA-CAMELLIA256-SHA256",
            "DHE-RSA-CHACHA20-POLY1305",
            "ECDHE-ARIA128-GCM-SHA256",
            "ECDHE-ARIA256-GCM-SHA384",
            "ECDHE-ECDSA-AES128-CCM",
            "ECDHE-ECDSA-AES128-CCM8",
            "ECDHE-ECDSA-AES128-GCM-SHA256",
            "ECDHE-ECDSA-AES128-SHA",
            "ECDHE-ECDSA-AES128-SHA256",
            "ECDHE-ECDSA-AES256-CCM",
            "ECDHE-ECDSA-AES256-CCM8",
            "ECDHE-ECDSA-AES256-GCM-SHA384",
            "ECDHE-ECDSA-AES256-SHA",
            "ECDHE-ECDSA-AES256-SHA384",
            "ECDHE-ECDSA-ARIA128-GCM-SHA256",
            "ECDHE-ECDSA-ARIA256-GCM-SHA384",
            "ECDHE-ECDSA-CAMELLIA128-SHA256",
            "ECDHE-ECDSA-CAMELLIA256-SHA384",
            "ECDHE-ECDSA-CHACHA20-POLY1305",
            "ECDHE-PSK-AES128-CBC-SHA",
            "ECDHE-PSK-AES128-CBC-SHA256",
            "ECDHE-PSK-AES256-CBC-SHA",
            "ECDHE-PSK-AES256-CBC-SHA384",
            "ECDHE-PSK-CAMELLIA128-SHA256",
            "ECDHE-PSK-CAMELLIA256-SHA384",
            "ECDHE-PSK-CHACHA20-POLY1305",
            "ECDHE-RSA-AES128-GCM-SHA256",
            "ECDHE-RSA-AES128-SHA",
            "ECDHE-RSA-AES128-SHA256",
            "ECDHE-RSA-AES256-GCM-SHA384",
            "ECDHE-RSA-AES256-SHA",
            "ECDHE-RSA-AES256-SHA384",
            "ECDHE-RSA-CAMELLIA128-SHA256",
            "ECDHE-RSA-CAMELLIA256-SHA384",
            "ECDHE-RSA-CHACHA20-POLY1305",
            "PSK-AES128-CBC-SHA",
            "PSK-AES128-CBC-SHA256",
            "PSK-AES128-CCM",
            "PSK-AES128-CCM8",
            "PSK-AES128-GCM-SHA256",
            "PSK-AES256-CBC-SHA",
            "PSK-AES256-CBC-SHA384",
            "PSK-AES256-CCM",
            "PSK-AES256-CCM8",
            "PSK-AES256-GCM-SHA384",
            "PSK-ARIA128-GCM-SHA256",
            "PSK-ARIA256-GCM-SHA384",
            "PSK-CAMELLIA128-SHA256",
            "PSK-CAMELLIA256-SHA384",
            "PSK-CHACHA20-POLY1305",
            "RSA-PSK-AES128-CBC-SHA",
            "RSA-PSK-AES128-CBC-SHA256",
            "RSA-PSK-AES128-GCM-SHA256",
            "RSA-PSK-AES256-CBC-SHA",
            "RSA-PSK-AES256-CBC-SHA384",
            "RSA-PSK-AES256-GCM-SHA384",
            "RSA-PSK-ARIA128-GCM-SHA256",
            "RSA-PSK-ARIA256-GCM-SHA384",
            "RSA-PSK-CAMELLIA128-SHA256",
            "RSA-PSK-CAMELLIA256-SHA384",
            "RSA-PSK-CHACHA20-POLY1305",
            "SRP-AES-128-CBC-SHA",
            "SRP-AES-256-CBC-SHA",
            "SRP-DSS-AES-128-CBC-SHA",
            "SRP-DSS-AES-256-CBC-SHA",
            "SRP-RSA-AES-128-CBC-SHA",
            "SRP-RSA-AES-256-CBC-SHA",
            "TLS_AES_128_GCM_SHA256",
            "TLS_AES_256_GCM_SHA384",
            "TLS_CHACHA20_POLY1305_SHA256"
        ]
    }
}

Debug Log

  • Add debug=True to create_scraper()
< GET /whiskies/springbank/hazelburn-sherry-8-year-old-whisky HTTP/1.1
< Host: www.masterofmalt.com
< User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
< Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
< Accept-Language: en-US,en;q=0.9
< Accept-Encoding: gzip, deflate
<

> HTTP/1.1 403 Forbidden
> Cache-Control: private, no-store, max-age=0
> Content-Type: text/html; charset=utf-8
> Server: Vercel
> X-Vercel-Challenge-Token: 2.1761144392.60.NzlkOGRmYjBmYmRkMzNlMDBkOTI3MGE3NmI1NmQyZDE7Nzc0ZDU2MTg7ZjcyMWYxMWM0NmIwNzc5N2NhZTlmNDZkYTBjMDZiM2Q3NDc4NGYxZTszOxNB7NEUYLTCNca4XpTIzJDEauEGizJTluLcxv5IZBtoFl1Q2wk4zLP6aOHKOtA0TVc=.38a8f31efa52a606ada17f334048a98f
> X-Vercel-Id: icn1::1761144392-9m9ExmVWciMffM1mcPy5twm3Q8vGpHHP
> X-Vercel-Mitigated: challenge
> Date: Wed, 22 Oct 2025 14:46:32 GMT
> Transfer-Encoding: chunked
>
<!DOCTYPE html><html lang="en" data-astro-cid-nbv56vs3> <head><meta charset="utf-8"><meta 
........
</div> </noscript>   </body></html>

This command is only available on cloudscraper v1.2.33 and greater. Otherwise, please provide some basic information about your system (Python version, operating system).

darkman13 avatar Oct 22 '25 14:10 darkman13

same here

datalith avatar Oct 27 '25 20:10 datalith

Same for me Any soultion?

ibrahimghareeb97 avatar Oct 28 '25 08:10 ibrahimghareeb97

Same, it was working until a couple of days, i migrate to this other tool: https://github.com/lexiforest/curl_cffi

Looks good and works for now

franfurey avatar Oct 29 '25 13:10 franfurey

Well, im creating kind of fallbacks, so if it fails with one library, it goes to the other.

And next finall step will be implement selenium or playwright, but they are heavier and slower, so i have that as final options

franfurey avatar Oct 29 '25 13:10 franfurey

Same for me -- cannot get passed the checkbox .. 403 is all that is returned except html of the failed challenge page. I am trying to reach a feed https://needtoknow.news/feed

HTTP/1.1 403 Forbidden

Date: Fri, 31 Oct 2025 12:42:32 GMT Content-Type: text/html; charset=UTF-8 Transfer-Encoding: chunked Connection: close accept-ch: Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA cf-mitigated: challenge critical-ch: Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA cross-origin-embedder-policy: require-corp cross-origin-opener-policy: same-origin cross-origin-resource-policy: same-origin origin-agent-cluster: ?1 permissions-policy: accelerometer=(),browsing-topics=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=() referrer-policy: same-origin server-timing: chlray;desc="99733ba3c92ebcdf" server-timing: cfOrigin;dur=0,cfEdge;dur=22 server-timing: cfL4;desc="?proto=TCP&rtt=41848&min_rtt=41128&rtt_var=15937&sent=5&recv=6&lost=0&retrans=0&sent_bytes=2886&recv_bytes=951&delivery_rate=98327&cwnd=252&unsent_bytes=0&cid=39080e8ef7ee1f15&ts=75&x=0" x-content-type-options: nosniff x-frame-options: SAMEORIGIN Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Expires: Thu, 01 Jan 1970 00:00:01 GMT Report-To: {"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=YCv8MHSn3EoT1zOWl3o0AJ8u%2BUnMT8PmiKC1jrD1oeIw2Qc%2FD15XAcHPALrjD2tToNohAI0hX7Lf2d8Ec2NGJCA%2FprJYrledJhgRJjh6O4uVmGrU4gis0TlLF27gKkTjyddW0Ia2xxkhqehcjXE%3D"}],"group":"cf-nel","max_age":604800} NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} Vary: Accept-Encoding Server: cloudflare CF-RAY: 99733ba3c92ebcdf-ATL Content-Encoding: gzip alt-svc: h3=":443"; ma=86400

steelliberty avatar Oct 31 '25 12:10 steelliberty

requests.exceptions.HTTPError: 403 Client Error: Forbidden for url:

flufy3d avatar Nov 06 '25 16:11 flufy3d

Not working anymore!

flufy3d avatar Nov 06 '25 16:11 flufy3d

@franfurey did you try it with https://github.com/0x676e67/rnet ? its much better maintance and new headers and tls.

JLFN avatar Nov 09 '25 19:11 JLFN

@franfurey did you try it with https://github.com/0x676e67/rnet ? its much better maintance and new headers and tls.

aa cool stuff! Thanks. No i didnt try it, for now with the curl_cffi, but when it starts falling i will add this one too. Thanks for sharing!

franfurey avatar Nov 10 '25 11:11 franfurey

@franfurey did you try it with https://github.com/0x676e67/rnet ? its much better maintance and new headers and tls.

Not working for CF JS challenge. Just tried on https;//1tv.ge and is not working

konkhra avatar Nov 23 '25 14:11 konkhra

Same, it was working until a couple of days, i migrate to this other tool: https://github.com/lexiforest/curl_cffi

Looks good and works for now

This is not going to work for CF JS challenge. Try on this https;//1tv.ge and see if is working. For me didn't worked.

konkhra avatar Nov 23 '25 14:11 konkhra