curse download fails again - possible cloudscraper fix

[ilu33]

Describe the bug All curse downloads fail with "403 None"

If you run pipenv update on the command line and try again, does the issue go away? No

Log output All curse downloads fail with "403 None" - no need to post the same log entry over and over.

I see that requests are made with HTTP/1.1. According to https://stackoverflow.com/questions/65604551/cant-bypass-cloudflare-with-python-cloudscraper it might help to

• simulate a browser as explained by dcts and
• use http2 like explained by Zorome

Sadly I'm not good enough to know where exactly to place these things in your code.

[ghost]

I just found a github user who mirrors curseforge addons. They also have a fork of cloudscraper with 34 new commits since the last update on upstream. This may be a solution, but I have not tried it. https://github.com/curseforge-mirror/cloudscraper

[DUTNO1]

It's the hardcoded UserAgent that's mucking things up, in updater/site/__init__.py change CURSE_UA from: CURSE_UA = {'custom': 'Chrome/80'} to: CURSE_UA = {'browser': 'chrome', 'platform': 'windows', 'mobile': False} - as mentioned in the SO thread and you should be good to go.

[grrttedwards]

@DUTNO1 Thanks. Yeah the user agent trick was a random discovery to fix Curse blocking years ago.

I may use Dragonflight as a reason to come and update this code. But it's tough for me to find the time. I appreciate everyone's dedication to this tool!

[ilu33]

@DUTNO1 Changing that did not help. Does it work for you?

@omicron-o Do you have any idea how one could include the modified library you mentioned instead of the on provided by pip? Is there a way to create a temporary local override to test it?

[DUTNO1]

@ilu33 welp, it seems like they "fixed" it - doesn't work for me anymore either.

[grrttedwards]

Looks like it's spontaneously working fine again:

Installing/updating addon: bagnon to version: 10.0.3...
...
Installing/updating addon: advancedinterfaceoptions to version: 1.8.1...

Name                        Prev. Version               New Version
────                        ─────────────               ───────────
hear-kitty                  -----                       1.10.0
claw                        -----                       9.0.2-6
...
deadly-boss-mods            -----                       DBM-10.0.0
Press any key to continue . . .

[ghost]

I suspect that sometimes it breaks because curse/cloudflare sometimes puts the site into a higher level of protection, causing some "v2" captcha to pop up that the library does not support. I'm not super sure though, just guessing based on log/exception output.

[arsirantala]

Would love that this issue is solved somehow - I checked where the addon files are actually downloaded (in curseforge's case), and they are hosted in mediafilez.forgecdn.net.

The url's suffix follows the addon name and its version.. I think it should be pretty straight forward to form a valid url in the script for the addon file from forgecdn to be downloaded (without headers/cookies been set).