cfscrape isn't working anymore (needs update)

[lifetalk]

Before creating an issue, first upgrade cfscrape with pip install -U cfscrape and see if you're still experiencing the problem. Please also confirm your Node version (node --version or nodejs --version) is version 10 or higher.

Make sure the website you're having issues with is actually using anti-bot protection by Cloudflare and not a competitor like Imperva Incapsula or Sucuri. And if you're using an anonymizing proxy, a VPN, or Tor, Cloudflare often flags those IPs and may block you or present you with a captcha as a result.

Please confirm the following statements and check the boxes before creating an issue:

• [/ ] I've upgraded cfscrape with pip install -U cfscrape
• [/ ] I'm using Node version 10 or higher
• [/ ] The site protection I'm having issues with is from Cloudflare
• [/ ] I'm not using Tor, a VPN, or an anonymizing proxy

Python version number

Run python --version and paste the output below:

Python 3.5.1

cfscrape version number

Run pip show cfscrape and paste the output below:

Name: cfscrape
Version: 2.1.1
Summary: A simple Python module to bypass Cloudflare's anti-bot page. See https://github.com/Anorov/cloudflare-scrape for more information.
Home-page: https://github.com/Anorov/cloudflare-scrape
Author: Anorov
Author-email: [email protected]
License: UNKNOWN
Location: /usr/local/lib/python3.5/site-packages
Requires: requests
Required-by:

Code snippet involved with the issue

import cfscrape

url = "https://ddl.to/login.html"
ua = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.86 Safari/537.36"
tokens, user_agent = cfscrape.get_cookie_string(url, user_agent=ua)
print (tokens)

Complete exception and traceback

(If the problem doesn't involve an exception being raised, leave this blank)

ERROR:root:Error executing Cloudflare IUAM Javascript. Cloudflare may have changed their technique, or there may be a bug in the script.

Please read https://github.com/Anorov/cloudflare-scrape#updates, then file a bug report at https://github.com/Anorov/cloudflare-scrape/issues."
ERROR:root:'https://ddl.to/login.html' returned an error. Could not collect tokens.
Traceback (most recent call last):
  File "/root/cf2", line 5, in <module>
    tokens, user_agent = cfscrape.get_cookie_string(url, user_agent=ua)
  File "/usr/local/lib/python3.5/site-packages/cfscrape/__init__.py", line 416, in get_cookie_string
    tokens, user_agent = cls.get_tokens(url, user_agent=user_agent, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/cfscrape/__init__.py", line 383, in get_tokens
    resp = scraper.get(url, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/cfscrape/__init__.py", line 129, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/cfscrape/__init__.py", line 204, in solve_cf_challenge
    answer, delay = self.solve_challenge(body, domain)
  File "/usr/local/lib/python3.5/site-packages/cfscrape/__init__.py", line 328, in solve_challenge
    raise subprocess.CalledProcessError(node.returncode, "node -e ...", stderr)
subprocess.CalledProcessError: Command 'node -e ...' returned non-zero exit status 8

URL of the Cloudflare-protected page

https://ddl.to/login.html

URL of Pastebin/Gist with HTML source of protected page

https://pastebin.com/GqhurQ7H

[NightMachinery]

I also get this error with the latest version:

Traceback (most recent call last):
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 255, in solve_challenge
    javascript, flags=re.S
AttributeError: 'NoneType' object has no attribute 'groups'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/evar/anaconda/lib/python3.7/site-packages/telethon/client/updates.py", line 428, in _dispatch_update
    await callback(event)
  File "stdplugins/torrentz.py", line 22, in _
    search_results = search_torrentz_eu(input_str)
  File "stdplugins/torrentz.py", line 77, in search_torrentz_eu
    raw_html = scraper.get(url).content
  File "/Users/evar/anaconda/lib/python3.7/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 129, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 219, in solve_cf_challenge
    redirect = self.request(method, submit_url, **cloudflare_kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 129, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 219, in solve_cf_challenge
    redirect = self.request(method, submit_url, **cloudflare_kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 129, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 219, in solve_cf_challenge
    redirect = self.request(method, submit_url, **cloudflare_kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 129, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 219, in solve_cf_challenge
    redirect = self.request(method, submit_url, **cloudflare_kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 129, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 243, in solve_cf_challenge
    resp = self.request(self.org_method, submit_url, **cloudflare_kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 129, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 204, in solve_cf_challenge
    answer, delay = self.solve_challenge(body, domain)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 292, in solve_challenge
    % BUG_REPORT
ValueError: Unable to identify Cloudflare IUAM Javascript on website. Cloudflare may have changed their technique, or there may be a bug in the script.

# pip show cfscrape
Name: cfscrape
Version: 2.1.1
Summary: A simple Python module to bypass Cloudflare's anti-bot page. See https://github.com/Anorov/cloudflare-scrape for more information.
Home-page: https://github.com/Anorov/cloudflare-scrape
Author: Anorov
Author-email: [email protected]
License: UNKNOWN
Location: /Users/evar/anaconda/lib/python3.7/site-packages
Requires: requests
Required-by:

# node -v
v12.7.0

[lrhys]

Hi guys, I faced errors as well when trying to get cfscrape working. I then found the exact same codebase uploaded on PyPi, named cloudscraper. Simply 'pip install cloudscraper' and use cloudscraper instead. It basically works the same.

[xyliutw]

Hi guys, I faced errors as well when trying to get cfscrape working. I then found the exact same codebase uploaded on PyPi, named cloudscraper. Simply 'pip install cloudscraper' and use cloudscraper instead. It basically works the same.

Thanks a lot @lrhys. It's work!! Thanks for your sharing!!

[wsch-wa]

Hi guys, I faced errors as well when trying to get cfscrape working. I then found the exact same codebase uploaded on PyPi, named cloudscraper. Simply 'pip install cloudscraper' and use cloudscraper instead. It basically works the same.

Thanks for your investigation. I changed to cloudscrapper and it worked. On weekend I did investigate. I think cfscrape fails because the "cf_clearance" token is empty.

[whoot]

@lrhys @xyliutw @wsch-wa Cloudscraper does not work anymore either...

[Sarfroz]

Hi guys, I faced errors as well when trying to get cfscrape working. I then found the exact same codebase uploaded on PyPi, named cloudscraper. Simply 'pip install cloudscraper' and use cloudscraper instead. It basically works the same.

thanks man it's working great.

[ArjixWasTaken]

cfscrape is really not working anymore, plz fix

[mandarraut2010]

Guys,its not working can someone please fix it ?

[AaronGeist]

I also get this error with the latest version:

Traceback (most recent call last):
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 255, in solve_challenge
    javascript, flags=re.S
AttributeError: 'NoneType' object has no attribute 'groups'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/evar/anaconda/lib/python3.7/site-packages/telethon/client/updates.py", line 428, in _dispatch_update
    await callback(event)
  File "stdplugins/torrentz.py", line 22, in _
    search_results = search_torrentz_eu(input_str)
  File "stdplugins/torrentz.py", line 77, in search_torrentz_eu
    raw_html = scraper.get(url).content
  File "/Users/evar/anaconda/lib/python3.7/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 129, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 219, in solve_cf_challenge
    redirect = self.request(method, submit_url, **cloudflare_kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 129, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 219, in solve_cf_challenge
    redirect = self.request(method, submit_url, **cloudflare_kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 129, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 219, in solve_cf_challenge
    redirect = self.request(method, submit_url, **cloudflare_kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 129, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 219, in solve_cf_challenge
    redirect = self.request(method, submit_url, **cloudflare_kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 129, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 243, in solve_cf_challenge
    resp = self.request(self.org_method, submit_url, **cloudflare_kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 129, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 204, in solve_cf_challenge
    answer, delay = self.solve_challenge(body, domain)
  File "/Users/evar/anaconda/lib/python3.7/site-packages/cfscrape/__init__.py", line 292, in solve_challenge
    % BUG_REPORT
ValueError: Unable to identify Cloudflare IUAM Javascript on website. Cloudflare may have changed their technique, or there may be a bug in the script.

# pip show cfscrape
Name: cfscrape
Version: 2.1.1
Summary: A simple Python module to bypass Cloudflare's anti-bot page. See https://github.com/Anorov/cloudflare-scrape for more information.
Home-page: https://github.com/Anorov/cloudflare-scrape
Author: Anorov
Author-email: [email protected]
License: UNKNOWN
Location: /Users/evar/anaconda/lib/python3.7/site-packages
Requires: requests
Required-by:

# node -v
v12.7.0

Same issue here. Looks like cloudflare has changed the script content so sfscrape cannot find correct content to continue working.

[isobarbaric]

I had the same issue as above and I tried using cloudscraper and it seems to be working.

[Soontosh]

The solution is to use zenrows.com, it's a proxy that fixed all of my issues(cloudflare didn't work for me either)

[ArjixWasTaken]

@Soontosh I think we all knew that a paid solution would work...