clj-http
clj-http copied to clipboard
Published
20 hours ago •
dakrone
dakrone
Compatibility issue with get request
When I make a request like so
(http/get "https://angel.co")
I get the response
1. Unhandled clojure.lang.ExceptionInfo
clj-http: status 403
{:cached nil,
:request-time 141,
:repeatable? false,
:protocol-version {:name "HTTP", :major 1, :minor 1},
:streaming? true,
:http-client
#object[org.apache.http.impl.client.InternalHttpClient 0x7d5d53f8 "org.apache.http.impl.client.InternalHttpClient@7d5d53f8"],
:chunked? true,
:type :clj-http.client/unexceptional-status,
:reason-phrase "Forbidden",
:headers
{"Server" "cloudflare",
"Content-Type" "text/html; charset=UTF-8",
"X-Frame-Options" "SAMEORIGIN",
"Connection" "close",
"cf-request-id" "02afa1c33a00000cb192b11200000001",
"Transfer-Encoding" "chunked",
"Set-Cookie"
"__cfduid=d5186984dc436aac8989538666ddb21761589373348; expires=Fri, 12-Jun-20 12:35:48 GMT; path=/; domain=.angel.co; HttpOnly; SameSite=Lax",
"Expect-CT"
"max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"",
"CF-RAY" "592c6be52bb00cb1-EWR",
"Date" "Wed, 13 May 2020 12:35:48 GMT",
"Vary" "Accept-Encoding",
"Cache-Control" "no-cache",
"CF-Chl-Bypass" "1"},
:orig-content-encoding nil,
:status 403,
:length -1,
:body
"<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]> <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 8]> <html class=\"no-js ie8 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->\n<head>\n<title>Attention Required! | Cloudflare</title>\n<meta name=\"captcha-bypass\" id=\"captcha-bypass\" />\n<meta charset=\"UTF-8\" />\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge,chrome=1\" />\n<meta name=\"robots\" content=\"noindex, nofollow\" />\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\" />\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\" type=\"text/css\" media=\"screen,projection\" />\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" type=\"text/css\" media=\"screen,projection\" /><![endif]-->\n<style type=\"text/css\">body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!--><script type=\"text/javascript\" src=\"/cdn-cgi/scripts/zepto.min.js\"></script><!--<![endif]-->\n<!--[if gte IE 10]><!--><script type=\"text/javascript\" src=\"/cdn-cgi/scripts/cf.common.js\"></script><!--<![endif]-->\n\n\n\n\n</head>\n<body>\n <div id=\"cf-wrapper\">\n <div class=\"cf-alert cf-alert-error cf-cookie-error\" id=\"cookie-alert\" data-translate=\"enable_cookies\">Please enable cookies.</div>\n <div id=\"cf-error-details\" class=\"cf-error-details-wrapper\">\n <div class=\"cf-wrapper cf-header cf-error-overview\">\n <h1 data-translate=\"challenge_headline\">One more step</h1>\n <h2 class=\"cf-subheadline\"><span data-translate=\"complete_sec_check\">Please complete the security check to access</span> angel.co</h2>\n </div><!-- /.header -->\n \n <div class=\"cf-section cf-highlight cf-captcha-container\">\n <div class=\"cf-wrapper\">\n <div class=\"cf-columns two\">\n <div class=\"cf-column\">\n \n <div class=\"cf-highlight-inverse cf-form-stacked\">\n <form class=\"challenge-form\" id=\"challenge-form\" action=\"/?__cf_chl_captcha_tk__=0d94e2e21d2ef06ef34bd2b5b4667f279b690108-1589373348-0-ATT4PiY_pQI1dGVw0_sZDV32_7x4mqtO4RepyD-L4i6zBJiIuml25fVlyJaK8uXNJw5ZWnzGlb6y0jGJJ8HIdEz14sOXRUoHqs_naHtwFEQywa8qZf_rwHsBxIUD5y_FNPph6TDrcfLVnQaN9eyy5VjiznzH4y0yeK8cidNnd-qNGw4OIZbFLfv8299DGhvNnBgsbn3BiQ9bkoGOtE4wANUh5U2LTJVAWhlquAvfhjCu6jHlYRXtN5GdnNvfBbCYwWGwCX0j88J-qCjJFOrSvx1_xraYtpB_Y8PpLHZTob_t8POfE0kJpn9ZYxwjhLQhqAAcIoE8fRe7Lv_50pzummklgMLgTRT2_NJGiE-_jNEogQmoTCvGOOmhNCe28SVYkXop9Ajm-z-6xwgoKQnY7EwekXJZCs-4nwpWJ9Gh3HBgVxZRiuv_wKgcmU0sPlLXSL5G8yOVdbBKBtHhQyqadtmTSg_IC2HV7SiYqPoJMmpJGfxxUm1au7ZS9ZiLpokjI5pQDZLpT2ZG-6jVfnTKvt9w_qmMtUSBhDleXd8mG59r\" method=\"POST\" enctype=\"application/x-www-form-urlencoded\">\n <input type=\"hidden\" name=\"r\" value=\"bfe8db4864c274e3ed80528a0e0ad233279c00b9-1589373348-0-AVWaRjujNq/XSmYrRyYxyBLhp5bbxA92rBX2qiiOx9PVWzas1b/usxApmblw248v1q5iUvP/V/GYHXhQF1UBviAqExhVjGW4upmNgdEf/zdFWHbgQb/s0RdZyMS+rurne8Y7aKD8ppx/WHjY8eSxVTGcHePc+qs/NdCt33voCLk2sGd0inuxibNjFXkBT62qs/JshlzaDsM58mC/jdSBRHZiOoJHmteJ0J1vwDVTVumWM97Qrc9fDyAqvDo72LCfqq0uG6hppWsi/z5jnGhTwzmJ7biqcY3BThvQAABSgD80MH4unfjys3iYhsefX0tfuAm23Rx1BCoKDRrrnWy0//Z9D0vI3petRmLSerLnJUAqCRh6ZoRqahYwNTPr39G+/WBJBsh3UDfB0+PwSmGsczRmL6DDbDu023etpAhehWcdR55ftEcijKiEnnfZE4vyKYm4C835QoKlQ+odT+u7syO/u/PgoyguQxqnNoKdlSSCs4+96s86urmY/yM9T4dvZdB4K4aOVH5cNfRHc8fsqeKpcuxmBbHOmIYIAegjTd5iKB4OQtxPHti1ZQCLeP74OiAxF6UgH+bCBp+h2mfU19CtEXvfcQdxGXPDT/iAPbPZG8c7fubDCKUympyb5nbHzVUcL9IGTlCq1zN7B1pRFj/O6JKOGBRo+q0OEs0nI7l/RFvmDfEtA0FYSC4IGegEs//fUsB165Zdm2SdKk7/cy89Xd4Hy5cedzqmjrtKNw5zjvfjqaNU7FlUL38irfopK/Pyk5Fp/HdV7iMvflIJO1M7GedTWdcNKB/OqPGV9NuJaKYgJbgBrxS4iYtHw9ZZsKWogYCig+eYiU8ty/MSDus9zCE2yRIbLVQ59AFwqTwODgBaV2nJepBDxcXVauCpdHiGbi7Q9M4t1eyGafFUKasv3unzdriRTrFPZ+44ZQb3gYberTMv2f3MwfcryaFgxcgtu43w8Hy5nviA9sOeoLmPYMZtL85QbB+AzKCXJV5DfIGcMvx1aeD/D9QNyOSTakVv2tAwxnP5UeQj8mJKGHTYrIsOMFDfxSnQ2lVzMRPQYmeEes8KjFvYrGyQ82Io+hGnKYOHX1T1ioi+wh+MGacVaSC1VMfG6rdIauPSxbB9WNxqnJxKz7SxHNiV3Gwm4rgUOs+vN2tSPyfINt12OHU=\">\n <input type=\"hidden\" name=\"cf_captcha_kind\" value=\"h\">\n <script type=\"text/javascript\" src=\"/cdn-cgi/scripts/hcaptcha.challenge.js\" data-type=\"normal\" data-ray=\"592c6be52bb00cb1\" async data-sitekey=\"33f96e6a-38cd-421b-bb68-7806e1764460\"></script>\n <noscript id=\"cf-captcha-bookmark\" class=\"cf-captcha-info\">\n <h1 data-translate=\"turn_on_js\" style=\"color:#bd2426;\">Please turn JavaScript on and reload the page.</h1>\n </noscript>\n <div id=\"trk_captcha_js\" style=\"background-image:url('/cdn-cgi/images/trace/captcha/nojs/h/transparent.gif?ray=592c6be52bb00cb1')\"></div>\n</form>\n\n </div>\n </div>\n\n <div class=\"cf-column\">\n <div class=\"cf-screenshot-container\">\n \n <span class=\"cf-no-screenshot\"></span>\n \n </div>\n </div>\n </div><!-- /.columns -->\n </div>\n </div><!-- /.captcha-container -->\n\n <div class=\"cf-section cf-wrapper\">\n <div class=\"cf-columns two\">\n <div class=\"cf-column\">\n <h2 data-translate=\"why_captcha_headline\">Why do I have to complete a CAPTCHA?</h2>\n \n <p data-translate=\"why_captcha_detail\">Completing the CAPTCHA proves you are a human and gives you temporary access to the web property.</p>\n </div>\n\n <div class=\"cf-column\">\n <h2 data-translate=\"resolve_captcha_headline\">What can I do to prevent this in the future?</h2>\n \n\n <p data-translate=\"resolve_captcha_antivirus\">If you are on a personal connection, like at home, you can run an anti-virus scan on your device to make sure it is not infected with malware.</p>\n\n <p data-translate=\"resolve_captcha_network\">If you are at an office or shared network, you can ask the network administrator to run a scan across the network looking for misconfigured or infected devices.</p>\n \n \n \n </div>\n </div>\n </div><!-- /.section -->\n \n\n <div class=\"cf-error-footer cf-wrapper\">\n <p>\n <span class=\"cf-footer-item\">Cloudflare Ray ID: <strong>592c6be52bb00cb1</strong></span>\n <span class=\"cf-footer-separator\">•</span>\n <span class=\"cf-footer-item\"><span>Your IP</span>: 128.151.150.1</span>\n <span class=\"cf-footer-separator\">•</span>\n <span class=\"cf-footer-item\"><span>Performance & security by</span> <a href=\"https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer\" id=\"brand_link\" target=\"_blank\">Cloudflare</a></span>\n \n </p>\n</div><!-- /.error-footer -->\n\n\n </div><!-- /#cf-error-details -->\n </div><!-- /#cf-wrapper -->\n\n <script type=\"text/javascript\">\n window._cf_translation = {};\n \n \n</script>\n\n\n</body>\n</html>\n",
:trace-redirects []}
support.clj: 201 slingshot.support/stack-trace
client.clj: 253 clj-http.client/exceptions-response
client.clj: 244 clj-http.client/exceptions-response
client.clj: 262 clj-http.client/wrap-exceptions/fn
client.clj: 734 clj-http.client/wrap-accept/fn
client.clj: 756 clj-http.client/wrap-accept-encoding/fn
client.clj: 717 clj-http.client/wrap-content-type/fn
client.clj: 958 clj-http.client/wrap-form-params/fn
client.clj: 992 clj-http.client/wrap-nested-params/fn
client.clj: 1016 clj-http.client/wrap-flatten-nested-params/fn
client.clj: 892 clj-http.client/wrap-method/fn
cookies.clj: 131 clj-http.cookies/wrap-cookies/fn
links.clj: 63 clj-http.links/wrap-links/fn
client.clj: 1045 clj-http.client/wrap-unknown-host/fn
client.clj: 1173 clj-http.client/request*
client.clj: 1166 clj-http.client/request*
client.clj: 1179 clj-http.client/get
client.clj: 1175 clj-http.client/get
RestFn.java: 410 clojure.lang.RestFn/invoke
REPL: 62 user/eval44577
REPL: 62 user/eval44577
Compiler.java: 7177 clojure.lang.Compiler/eval
Compiler.java: 7132 clojure.lang.Compiler/eval
core.clj: 3214 clojure.core/eval
core.clj: 3210 clojure.core/eval
interruptible_eval.clj: 91 nrepl.middleware.interruptible-eval/evaluate/fn
main.clj: 437 clojure.main/repl/read-eval-print/fn
main.clj: 437 clojure.main/repl/read-eval-print
main.clj: 458 clojure.main/repl/fn
main.clj: 458 clojure.main/repl
main.clj: 368 clojure.main/repl
RestFn.java: 137 clojure.lang.RestFn/applyTo
core.clj: 665 clojure.core/apply
core.clj: 660 clojure.core/apply
regrow.clj: 18 refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
RestFn.java: 1523 clojure.lang.RestFn/invoke
interruptible_eval.clj: 84 nrepl.middleware.interruptible-eval/evaluate
interruptible_eval.clj: 56 nrepl.middleware.interruptible-eval/evaluate
interruptible_eval.clj: 155 nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
AFn.java: 22 clojure.lang.AFn/run
session.clj: 190 nrepl.middleware.session/session-exec/main-loop/fn
session.clj: 189 nrepl.middleware.session/session-exec/main-loop
AFn.java: 22 clojure.lang.AFn/run
Thread.java: 748 java.lang.Thread/run
Where the body says <html class="no-js ie6 oldie" lang="en-US">. How do I fix this? It's supposed to return the actual webpage?
It looks like the page is detecting that javascript is not enabled (because it's not!), and then presenting a captcha, in the meantime though it's returning a 403. You could try changing the :user-agent, but unfortunately, it's the page that's behaving poorly, not clj-http.
