cloudscraper
cloudscraper copied to clipboard
codemanki
1020, Access Denied (Custom Firewall Rules)
I cannot access https://www.mabbank.com/ today. my server is located in Singapore. until today, it's working well.
REQUEST { requester:
{ [Function: request]
get: [Function],
head: [Function],
options: [Function],
post: [Function],
put: [Function],
patch: [Function],
del: [Function],
delete: [Function],
jar: [Function],
cookie: [Function],
defaults: [Function],
forever: [Function],
Request:
{ [Function: Request]
super_: [Function],
debug: true,
defaultProxyHeaderWhiteList: [Array],
defaultProxyHeaderExclusiveList: [Array] },
initParams: [Function: initParams],
debug: [Getter/Setter],
bindCLS: [Function: RP$bindCLS] },
jar:
RequestJar {
_jar: CookieJar { enableLooseMode: true, store: { idx: {} } } },
headers:
{ Host: Symbol(host),
Connection: 'keep-alive',
'Upgrade-Insecure-Requests': '1',
'User-Agent':
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36',
Accept:
'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.8',
'Accept-Encoding': 'gzip, deflate, sdch, br' },
cloudflareMaxTimeout: 30000,
followAllRedirects: true,
challengesToSolve: 3,
decodeEmails: false,
gzip: true,
agentOptions:
{ ciphers:
'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!SRP:!CAMELLIA:!ECDHE+SHA:!AES128-SHA' },
uri: 'https://www.mabbank.com/',
method: 'GET',
realEncoding: 'utf8',
encoding: null,
callback: [Function: RP$callback],
transform: undefined,
simple: true,
resolveWithFullResponse: false,
transform2xxOnly: false }
REQUEST make request https://www.mabbank.com/
REQUEST onRequestResponse https://www.mabbank.com/ 503 { date: 'Wed, 22 Apr 2020 05:46:48 GMT',
'content-type': 'text/html; charset=UTF-8',
'transfer-encoding': 'chunked',
connection: 'close',
'x-frame-options': 'SAMEORIGIN',
'set-cookie':
[ '__cfduid=dcb46a322884336e07ec1218bb44d86821587534408; expires=Fri, 22-May-20 05:46:48 GMT; path=/; domain=.mabbank.com; HttpOnly; SameSite=Lax' ],
'cache-control': 'no-cache',
'expect-ct':
'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"',
vary: 'Accept-Encoding',
'strict-transport-security': 'max-age=31536000; includeSubDomains; preload',
'x-content-type-options': 'nosniff',
server: 'cloudflare',
'cf-ray': '587d0be68ee7e233-SIN',
'cf-request-id': '024205c4130000e23390a7c200000001' }
REQUEST reading response's body
REQUEST finish init function https://www.mabbank.com/
REQUEST response end https://www.mabbank.com/ 503 { date: 'Wed, 22 Apr 2020 05:46:48 GMT',
'content-type': 'text/html; charset=UTF-8',
'transfer-encoding': 'chunked',
connection: 'close',
'x-frame-options': 'SAMEORIGIN',
'set-cookie':
[ '__cfduid=dcb46a322884336e07ec1218bb44d86821587534408; expires=Fri, 22-May-20 05:46:48 GMT; path=/; domain=.mabbank.com; HttpOnly; SameSite=Lax' ],
'cache-control': 'no-cache',
'expect-ct':
'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"',
vary: 'Accept-Encoding',
'strict-transport-security': 'max-age=31536000; includeSubDomains; preload',
'x-content-type-options': 'nosniff',
server: 'cloudflare',
'cf-ray': '587d0be68ee7e233-SIN',
'cf-request-id': '024205c4130000e23390a7c200000001' }
REQUEST end event https://www.mabbank.com/
REQUEST has body https://www.mabbank.com/ 11151
REQUEST emitting complete https://www.mabbank.com/
{ CloudflareError: 1020, Access Denied (Custom Firewall Rules)
at validateResponse (/opt/app/node_modules/cloudscraper/index.js:281:11)
at onCloudflareResponse (/opt/app/node_modules/cloudscraper/index.js:222:5)
at onRequestResponse (/opt/app/node_modules/cloudscraper/index.js:205:5)
at Request.<anonymous> (/opt/app/node_modules/cloudscraper/index.js:149:7)
at Object.onceWrapper (events.js:286:20)
at Request.emit (events.js:198:13)
at Request.EventEmitter.emit (domain.js:448:20)
at Request.<anonymous> (/opt/app/node_modules/request/request.js:1154:10)
at Request.emit (events.js:198:13)
at Request.EventEmitter.emit (domain.js:448:20)
at IncomingMessage.<anonymous> (/opt/app/node_modules/request/request.js:1076:12)
at Object.onceWrapper (events.js:286:20)
at IncomingMessage.emit (events.js:203:15)
at IncomingMessage.EventEmitter.emit (domain.js:448:20)
at endReadableNT (_stream_readable.js:1143:12)
at process._tickCallback (internal/process/next_tick.js:63:19)
name: 'CloudflareError',
message: '1020, Access Denied (Custom Firewall Rules)' }
Same issue here, on multiple sites. Works in browser, but not through cloudscraper Changing the TLS ciphers as suggested in other issues seems to have no effect.
CloudflareError: 1020, Access Denied (Custom Firewall Rules) at validateResponse (/root/js/cloudscraper/index.js:281:11) at onCloudflareResponse (/root/js/cloudscraper/index.js:222:5) at onRequestResponse (/root/js/cloudscraper/index.js:205:5) at Request.<anonymous> (/root/js/cloudscraper/index.js:149:7) at Object.onceWrapper (events.js:422:26) at Request.emit (events.js:315:20) at Request.<anonymous> (/root/js/node_modules/request/request.js:1161:10) at Request.emit (events.js:315:20) at IncomingMessage.<anonymous> (/root/js/node_modules/request/request.js:1083:12)
same problem. Yesterday I saw on the cloudflare website that it said "update"
Have the same problem. Sometimes also have next: { ParserError:
Cloudflare may have changed their technique, or there may be a bug.
Bug Reports: https://github.com/codemanki/cloudscraper/issues
Check the detailed exception message that follows for the cause.
challengeId (jschl_vc) extraction failed at onChallenge (/var/www/bt/node_modules/cloudscraper/index.js:317:21) at onCloudflareResponse (/var/www/bt/node_modules/cloudscraper/index.js:247:12) ...
Thats because script trying to find match = body.match(/name="jschl_vc" value="(\w+)"/);
But if I look at html code I see <input type="hidden" value="6ea98459f78d801733a08c06b2a090ca" id="jschl-vc" name="jschl_vc"/>
Field VALUE located before NAME and ID between them
Other cloudflare scrapers have this problem too https://github.com/VeNoMouS/cloudscraper/issues/185 https://github.com/VeNoMouS/cloudscraper/issues/197
Guys, can you check if this fix works for you?
https://github.com/bestplay9384/cloudscraper/commit/a8d45e97a81b856160f94e1d96114d4498e2fffb
For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/
Guys, can you check if this fix works for you?
For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/
yes, this works for me, thanks
Guys, can you check if this fix works for you?
For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/
@bestplay9384 Great! Your patch works well for me as well.
The original code looks a bit unstable to extract values from node. I appreciate the way you are doing in. An ultimate solution is probably using a DOM parser like cheerio to find and extract value from a node.
We should think about replacing the cloudscraper, it seems to be dead, no one has dealt with updates for 2 months, although several errors have been reported
Guys, can you check if this fix works for you?
For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/
Don't work for me.
I haven't setTimeout(function(){ ... },4000);
Haven't all challenge code. Instead I have the new script:
window._cf_chl_opt={
cvId: "1",
cType: "non-interactive",
cNounce: "55205",
cRay: "5885476c2b20f981",
cHash: "b6325db0634ad7e",
cRq: {
d: "i/e6L9mNU0Wx4EEDXDKILOHqqIeQZHqdTnud0VUS6oEak4TnVqioSS2PKNOBpUnvBkcUVKuYNRaOX1uth0J77BagcXJfekfI/1DJiu6tSQvzkIR1ZG9XPLFAdDhcVkFnrtkNd+ynKoNcCHGB6QccWxynrymxFijfoT6bo4oULfP8G0MH5OsGWzAMMs3/IZs6VTA1q9GtKyv3ZvAe+dlV6GLR+50miyDsoKIZeLsAIH9HjhS7fGRegId/PyOK15LjiQdhg1yTDkfl3J8cumCgiFCan/p25GhmJUkh3985yB74FEGzDOTlUR3Sm5MqLwBL",
t: "MTU4NzYyMDczMi44MjkwMDA=",
m: "PaC2ctNpbo+tHnNEsBhyftv81KgQpM6Pi/M7oPDDDEM=",
i1: "SOpF4po/BqGPmVAiq7vIqw==",
i2: "2MtEnwHnzWOUFN04e//L3w==",
}
}
window._cf_chl_enter = function(){window._cf_chl_opt.p=1};
var a = function() {try{return !!window.addEventListener} catch(e) {return !1} },
b = function(b, c) {a() ? document.addEventListener("DOMContentLoaded", b, c) : document.attachEvent("onreadystatechange", b)};
b(function(){
var a = document.getElementById('cf-content');a.style.display = 'block';
var isIE = /(MSIE|Trident\/|Edge\/)/i.test(window.navigator.userAgent);
var trkjs = isIE ? new Image() : document.createElement('img');
trkjs.setAttribute("src", "/cdn-cgi/images/trace/jschal/js/transparent.gif?ray=5885476c2b20f981");
trkjs.id = "trk_jschal_js";
trkjs.setAttribute("alt", "");
document.body.appendChild(trkjs);
document.body.appendChild(trkjs);
var cpo=document.createElement('script');
cpo.type='text/javascript';
cpo.src="/cdn-cgi/challenge-platform/orchestrate/jsch/v1";
cpo.onload=function(){window._cf_chl_enter()};
document.head.appendChild(cpo);
}, false);
@Dima22 can you provide an URL you was requesting? Or at last the whole HTML of the website?
@bestplay9384 https://betscsgo.cc/ Maybe this script runs just once for IP. Because now in my browser I have challenge code. Or maybe it runs periodically
@Dima22 yea it looks like that it is something new CF provided. I was finally able to see this protection while hard-refreshing view-source:https://betscsgo.cc/ in incognito mode multiple times :) I will try to come up with a way of bypassing this, however as far as i can see eg. here: https://github.com/VeNoMouS/cloudscraper/commit/72f2204b36532242f7cf65510f4bb5602cf8ed21, others are just detecting this "new way". I'll give it a try if i find some time!
Guys, can you check if this fix works for you?
For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/
This work for me also... thanks!
Guys, can you check if this fix works for you? bestplay9384@a8d45e9 For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/
@bestplay9384 Great! Your patch works well for me as well.
The original code looks a bit unstable to extract values from node. I appreciate the way you are doing in. An ultimate solution is probably using a DOM parser like
cheerioto find and extract value from a node.
@stanleyxu2005 i have found some time gap - all the stuff with DOM parsing are now using cheerio as you suggested. You can take a look on my forked repo :) I think there is still something missing in this bypass - probably no new hCaptcha detection & it is still not detecting the new challengeUI @Dima22 provided (i think they are just testing it randomly, because i can get it in 1 per 30 requests...) and it is still not able to bypass the new UI :)
Guys, can you check if this fix works for you?
For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/
same error using this new version, 2 times working and now again ask for captcha
if I try 4 times in a row, it works 2 times, and then it asks for captcha
url tested: https://cyberwarblog.xyz/
// on this url also dont work https://fatality.win/
Guys, can you check if this fix works for you?
For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/
Thanks a lot, works like a charm. If this repo drops would you consider picking it up? I really don't want to use Selenium etc ;(
Guys, can you check if this fix works for you? bestplay9384@a8d45e9 For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/
Thanks a lot, works like a charm. If this repo drops would you consider picking it up? I really don't want to use Selenium etc ;(
Just work some times :D if u are testing many times in same url will get same error
for me work 2 times, and if i try again in same url i get captcha error
for ex i have tested on https://cyberwarblog.xyz/ https://fatality.win/
I try 10 times, 4 times it works, and 6 times I get a captcha error in the same url, using different servers, without using proxies
It may not be the best way to do it, but it worked for me.
` const data = {}; let inputs = body.match(/(<input(.*)/>)/g);
for (const input of inputs) {
const inputName = input.match(/(name=\"\w+\")/)[0].slice(6, -1);
switch (inputName) {
case "r":
case "jschl_vc":
case "pass":
const inputValue = input
.match(/(value=\"(.[^"]+)\")/)[0]
.slice(7, -1);
data[inputName] = inputValue;
break;
}
}
`
Hi guys, Has anyone solved this problem? Cloudscraper is no longer maintained?
Shit. After 3 days of debugging and strained eyes, I have finally cracked it. Cloudscraper method of cracking the answer is not really efficient or accurate. I will post a new repo soon, already written most part and it's based on got.
Added repository @sayem314/hooman
TODO
- Solve hCaptcha
Test are done on https://cf-js-challenge.sayem.eu.org
@sayem314 This will only works for the old challenge (who only do jsFuck and basic DOM validation) but not for the new challenge they added in april. I don't know if they started to release it worldwide yet.
Also, i don't think it's a good idea to run arbitrary code in Eval. A malicious server could send any JS code and take control of the process/server. You should at minimum use a sandbox and parser to limit or remove malicious code.
@francisvalois I see. Then probably those challenges are not available for me yet. And about eval yes it should be run inside sandbox but again hooman is meant for accessing known sites but still I will try to safe it on next version :) Thanks.
If anyone has new challenge HTML from browser could you please share here? Please upload to https://paste.ubuntu.com/
Edit: code are now run safely using vm2. Commit https://github.com/sayem314/hooman/commit/7332cc2d40c0be3608e6a91718831f5ed2f6f896
@sayem314 your script does not work with yggtorrent it leave me with a 403 response code (Forbidden)
Guys, can you check if this fix works for you?
For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/
Nice fix, thanks. The only problem I see is that console.log(stringBody); line keeps terminal window spammed with unnecessary html
