Doesn't seem to work with CloudFlare?

[thistehneisen]

➜  Bash shcheck.py https://www.printful.com/

======================================================
 > shcheck.py - santoru ..............................
------------------------------------------------------
 Simple tool to check security headers on a webserver
======================================================

[!] URL Returned an HTTP error: 503
Target host seems to be unreachable (Service Temporarily Unavailable)
➜  Bash shcheck.py https://www.printful.com/

======================================================
 > shcheck.py - santoru ..............................
------------------------------------------------------
 Simple tool to check security headers on a webserver
======================================================

[!] URL Returned an HTTP error: 503
Target host seems to be unreachable (Service Temporarily Unavailable)
➜  Bash shcheck.py https://www.printful.com/ -p 443

======================================================
 > shcheck.py - santoru ..............................
------------------------------------------------------
 Simple tool to check security headers on a webserver
======================================================

[!] URL Returned an HTTP error: 503
Target host seems to be unreachable (Service Temporarily Unavailable)
➜  Bash shcheck.py https://printful.com

======================================================
 > shcheck.py - santoru ..............................
------------------------------------------------------
 Simple tool to check security headers on a webserver
======================================================

[!] URL Returned an HTTP error: 503
Target host seems to be unreachable (Service Temporarily Unavailable)
➜  Bash shcheck.py https://www.printful.com

======================================================
 > shcheck.py - santoru ..............................
------------------------------------------------------
 Simple tool to check security headers on a webserver
======================================================

[!] URL Returned an HTTP error: 503
Target host seems to be unreachable (Service Temporarily Unavailable)
➜  Bash shcheck.py http://printful.com

======================================================
 > shcheck.py - santoru ..............................
------------------------------------------------------
 Simple tool to check security headers on a webserver
======================================================

[!] URL Returned an HTTP error: 503
Target host seems to be unreachable (Service Temporarily Unavailable)
➜  Bash shcheck.py https://www.printful.com -d

======================================================
 > shcheck.py - santoru ..............................
------------------------------------------------------
 Simple tool to check security headers on a webserver
======================================================

[!] URL Returned an HTTP error: 503
Target host seems to be unreachable (Service Temporarily Unavailable)

[santoru]

Hi, thank you for reporting this. Interesting. According to CloudFlare:

The Error 503: Service Temporarily Unavailable (with “Cloudflare” visible in browser) means you are hitting a connection limit in a Cloudflare data center. When this happens, please contact Cloudflare Customer Support.

Probably it's some kind of rate limit on their side, I was able to do bunch of successful requests:

 > shcheck.py - santoru ..............................
------------------------------------------------------
 Simple tool to check security headers on a webserver
======================================================

[*] Analyzing headers of https://www.printful.com
[*] Effective URL: https://www.printful.com
[!] Missing security header: X-Frame-Options
[*] Header X-Content-Type-Options is present! (Value: nosniff)
[*] Header Strict-Transport-Security is present! (Value: max-age=31536000; includeSubDomains; preload)
[!] Missing security header: Content-Security-Policy
[!] Missing security header: Referrer-Policy
[*] Header Expect-CT is present! (Value: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct")
[!] Missing security header: Permissions-Policy
[!] Missing security header: Cross-Origin-Embedder-Policy
[!] Missing security header: Cross-Origin-Resource-Policy
[!] Missing security header: Cross-Origin-Opener-Policy
-------------------------------------------------------
[!] Headers analyzed for https://www.printful.com
[+] There are 3 security headers
[-] There are not 9 security headers

Can you confirm it worked for you as well at least once? Anyway I'm closing this as it's mostly a limit on CF side rather than a bug. Thank you.

[santoru]

My bad for closing this too early :D Seems to be a CF protection for bot/scanners/headless browsers. I'll keep this open and I'll investigate if it can be easily fixed

[thistehneisen]

I haven't reviewed much of the source, but if I'd have to guess, CF probably is showing the captcha page here and not allowing to get any further. I did succeed with the test when the IP address from which the script is ran - is whitelisted. If that's the case, it's probably not solvable easily, as you'd have to emulate a real browser (chromium driver etc.).

[SepehrRasouli]

Could we solve this problem by using an webdriver if we ever hit a 503 HTTP error ? I think it might be solvable through this or at least letting the user know that they've reached a limit.

[danielklecha]

Cloudflare could return 503 when "Bot Fight Mode" or "Super Bot Fight Mode" is enabled. Cloudflare doesn't block Verified Bots. User cannot disable this module in the Free plan.

I think that usage of Webdriver goes beyond simple checking. Also, I assume that if shcheck got 503 then IP could be blocked and Webdriver will be blocked too.