ServerError: 503 Server Error: Service Temporarily Unavailable

[orel35]

Traceback (most recent call last): File "C:\perso\tools\docto\doctoshotgun.py", line 506, in sys.exit(Application().main()) File "C:\perso\tools\docto\doctoshotgun.py", line 433, in main if not docto.do_login(): File "C:\perso\tools\docto\doctoshotgun.py", line 192, in do_login self.open('https://www.doctolib.fr/sessions/new') File "C:\Users\aurel\AppData\Local\Programs\Python\Python39\lib\site-packages\woob\browser\browsers.py", line 898, in open return super(PagesBrowser, self).open(callback=internal_callback, *args, **kwargs) File "C:\Users\aurel\AppData\Local\Programs\Python\Python39\lib\site-packages\woob\browser\browsers.py", line 790, in open return super(DomainBrowser, self).open(req, *args, **kwargs) File "C:\Users\aurel\AppData\Local\Programs\Python\Python39\lib\site-packages\woob\browser\browsers.py", line 531, in open response = self.session.send(preq, File "C:\perso\tools\docto\doctoshotgun.py", line 57, in send return callback(self, resp) File "C:\Users\aurel\AppData\Local\Programs\Python\Python39\lib\site-packages\woob\browser\browsers.py", line 527, in inner_callback self.raise_for_status(response) File "C:\Users\aurel\AppData\Local\Programs\Python\Python39\lib\site-packages\woob\browser\browsers.py", line 563, in raise_for_status raise ServerError(http_error_msg, response=response) woob.browser.exceptions.ServerError: 503 Server Error: Service Temporarily Unavailable

[nicolaus-hee]

Do you get this consistently? Usually, trying again will 'resolve' it.

Possibly related to #31 , #21

[rbignon]

I can't reproduce.

[sgoendoer]

Hi, I started to get this also. First from time to time, then permanently. My assumption is that the guys at Doctolib figured that someone is using a (this) script and started blocking it. Not sure if this generally detects this script and blocks it or if this is an IP-based blocking.

[porlk]

I can confirm to have had the same issue. In my case, the casing of the city name was the cause (eg. you should write paris or grenoble instead of Paris and Grenoble).

[gitolicious]

I tried to learn a bit more about this error. For me it's CloudFlare blocking my requests.

    def do_login(self):
        while True:
            retryAfterLoginError = False
            try:
                self.open(self.BASEURL + '/sessions/new')
            except ServerError as e:
                if e.response.status_code in [503]:
                    if 'text/html' in e.response.headers['Content-Type'] and 'cloudflare' in e.response.text:
                        log('Request blocked by CloudFlare', color='red')
                    log('Login failed. Retrying in 10s...')
                    self.session.close()
                    retryAfterLoginError = True
                    sleep(10)
                else:
                    raise
            if retryAfterLoginError == False:
                break

With this change, the login is retried though I didn't have any success even after multiple retries.

$ ./doctoshotgun.py de koln [email protected] password --dry-run
Request blocked by CloudFlare
Login failed. Will retry in 10 sec
Request blocked by CloudFlare
Login failed. Will retry in 10 sec

Stopping and starting the script immediately after helps sometimes though. This is weird behavior.

$ ./doctoshotgun.py de koln [email protected] password --dry-run
Request blocked by CloudFlare
Login failed. Will retry in 10 sec
^CAbort.
$ ./doctoshotgun.py de koln [email protected] password --dry-run
Starting to look for vaccine slots for John Doe between 2021-06-12 and 2021-06-19...
Vaccines: Pfizer, Moderna, Janssen
Country: de
This may take a few minutes/hours, be patient!

[rbignon]

That's strange, I never experienced this… Do you use the latest version of cloudscraper, and do you run your script from a home ISP (not from a server)?

[nicarl]

I saw the problem only when running from a VPS and after some days (the IP address of the VPS never changes).

[rbignon]

Yes, I guess it is harder to bypass cloudflare from a server.

[gitolicious]

I was running the script inside a Docker container. When doing so on my laptop (VS Code Dev Container as well as Alpine base image) I got 503s from time to time (~2 out of 10), when doing so on my Odroid N2+ smart home central (Linux OS without a Desktop) it happens most of the time (~9 out of 10). Seems like cloudscaper doesn't play nice with Non-UI environments and that's something doctoshotgun is not to blame for.

[gitolicious]

After playing around more with the script, in my Docker environments it is a hit and miss. Once it is running, it works fine all the way - if it starts with a Cloudflare error this seems not to be recoverable with waiting loops. I will add the more detailed error message but not the retries.

[lurkaround]

I never get past this error running the script on my local machine. Per #21, I don't guess this has been fixed yet?

[gitolicious]

Are you blocked by the Cloudflare check or what response are you getting?

[HelonDCI]

Whoops, probably should have included this initially. Here's the log: Traceback (most recent call last): File "C:\Users\tools\doctoshotgun\doctoshotgun.py", line 869, in <module> sys.exit(Application().main()) File "C:\Users\tools\doctoshotgun\doctoshotgun.py", line 698, in main if not docto.do_login(): File "C:\Users\tools\doctoshotgun\doctoshotgun.py", line 216, in do_login self.open(self.BASEURL + "/sessions/new") File "C:\Python39\lib\site-packages\woob\browser\browsers.py", line 898, in open return super(PagesBrowser, self).open(callback=internal_callback, *args, **kwargs) File "C:\Python39\lib\site-packages\woob\browser\browsers.py", line 790, in open return super(DomainBrowser, self).open(req, *args, **kwargs) File "C:\Python39\lib\site-packages\woob\browser\browsers.py", line 531, in open response = self.session.send(preq, File "C:\Users\tools\doctoshotgun\doctoshotgun.py", line 64, in send return callback(self, resp) File "C:\Python39\lib\site-packages\woob\browser\browsers.py", line 527, in inner_callback self.raise_for_status(response) File "C:\Python39\lib\site-packages\woob\browser\browsers.py", line 563, in raise_for_status raise ServerError(http_error_msg, response=response) woob.browser.exceptions.ServerError: 503 Server Error: Service Temporarily Unavailable

[gitolicious]

I was able to reproduce this in a Docker environment. This is what I see in the HTML response:

Checking your browser before accessing doctolib.de.
This process is automatic. Your browser will redirect to your requested content shortly.
Please allow up to 5 seconds
Redirecting

So this is a Cloudflare check that blocks the request which is not identified by the changes in #74. (More info on this "Dynamic Analysis" can be found here)

I am unsure if this means cloudscraper has ultimately failed to circumvent the Cloudflare checks, or it will handle it once we raise the error.

[gitolicious]

We now have explanatory error messages and nothing else we can do other than retrying or hoping for cloudscraper to improve.

[tobiasBora]

I can also confirm this, with the enhanced cloudflare log message PR I get a similar error Request blocked by CloudFlare. This was run locally.

Request blocked by CloudFlare
Traceback (most recent call last):
  File "/tmp/nix-shell.DpTZUZ/tmp.IIan1yyEqe/bin/doctoshotgun.py", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/me/doctoshotgun/doctoshotgun.py", line 863, in <module>
    sys.exit(Application().main())
  File "/home/me/doctoshotgun/doctoshotgun.py", line 686, in main
    if not docto.do_login(args.code):
  File "/home/me/doctoshotgun/doctoshotgun.py", line 259, in do_login
    self.open(self.BASEURL + '/sessions/new')
  File "/nix/store/vjz7gf1v4jicn88v2m6a7knyshzqbv25-python3.8-woob-3.0/lib/python3.8/site-packages/woob/browser/browsers.py", line 898, in open
    return super(PagesBrowser, self).open(callback=internal_callback, *args, **kwargs)
  File "/nix/store/vjz7gf1v4jicn88v2m6a7knyshzqbv25-python3.8-woob-3.0/lib/python3.8/site-packages/woob/browser/browsers.py", line 790, in open
    return super(DomainBrowser, self).open(req, *args, **kwargs)
  File "/nix/store/vjz7gf1v4jicn88v2m6a7knyshzqbv25-python3.8-woob-3.0/lib/python3.8/site-packages/woob/browser/browsers.py", line 531, in open
    response = self.session.send(preq,
  File "/home/me/doctoshotgun/doctoshotgun.py", line 74, in send
    return callback(self, resp)
  File "/nix/store/vjz7gf1v4jicn88v2m6a7knyshzqbv25-python3.8-woob-3.0/lib/python3.8/site-packages/woob/browser/browsers.py", line 527, in inner_callback
    self.raise_for_status(response)
  File "/nix/store/vjz7gf1v4jicn88v2m6a7knyshzqbv25-python3.8-woob-3.0/lib/python3.8/site-packages/woob/browser/browsers.py", line 563, in raise_for_status
    raise ServerError(http_error_msg, response=response)
woob.browser.exceptions.ServerError: 503 Server Error: Service Temporarily Unavailable

[gitolicious]

Did it work for you when retrying right away?

[flrnt]

same error here, tried using a proxy (germany, spain, argentina, belgium, whatever..) but none worked

[Stephan972]

I get stuck with the very same error. I'm connecting from my home (ie no VPS). Any try is a complete failure (even with large pause between two attempts).