auto-southwest-check-in
auto-southwest-check-in copied to clipboard
jdholtz
Forbidden 403
Version
develop tag Auto-Southwest Check-In v7.2
Browser Version
Using browser version: 121.0.6167.184
Description
I'm getting consistent 403 errors when Retrieving reservation information. I purposely limited my config down to a single record locator.. removing the account login stuff and setting check_fares to false. I updated to the latest develop tag just now as well. I ran it 4-5 times in a row and got the same result each time.
To Reproduce
Running it via docker-compose w/ config.json w/ a single record locator.
Expected Behavior
I'd expect it to schedule a checkin at minimum, and ideally successfully check in at that scheduled time.
Relevant logs and program output
my config.json
{
"check_fares": false,
"notification_urls": "gchat://XXXXXXX",
"notification_level": 1,
"retrieval_interval": 6,
"reservations": [
{"confirmationNumber": "XXXXXX", "firstName": "XXXXXX", "lastName": "XXXXXXXXXX"}
]
}
docker logs output:
2024-02-26 15:23:21 DEBUG MainProcess[log:23]: Initialized the application
2024-02-26 15:23:21 DEBUG MainProcess[main:112]: Auto-Southwest Check-In v7.2
2024-02-26 15:23:21 DEBUG MainProcess[main:113]: Called with 0 arguments
2024-02-26 15:23:21 DEBUG MainProcess[config:120]: Initializing configuration file
2024-02-26 15:23:21 DEBUG MainProcess[config:149]: Reading the configuration file
2024-02-26 15:23:21 DEBUG MainProcess[config:163]: Reading configuration from environment variables
2024-02-26 15:23:21 DEBUG MainProcess[config:58]: Setting check fares to False
2024-02-26 15:23:21 DEBUG MainProcess[config:80]: Setting notification level to <NotificationLevel.INFO: 1>
2024-02-26 15:23:21 DEBUG MainProcess[config:93]: Using 1 notification services
2024-02-26 15:23:21 DEBUG MainProcess[config:97]: Setting retrieval interval to 6 hours
2024-02-26 15:23:21 DEBUG MainProcess[config:139]: Creating configurations for 1 reservations
2024-02-26 15:23:21 DEBUG MainProcess[main:142]: Monitoring 0 accounts and 1 reservations
2024-02-26 15:23:21 DEBUG Process-1[reservation_monitor:60]: Acquiring lock...
2024-02-26 15:23:21 DEBUG Process-1[reservation_monitor:62]: Lock acquired
2024-02-26 15:23:21 DEBUG Process-1[checkin_scheduler:50]: Refreshing headers for current session
2024-02-26 15:23:21 DEBUG Process-1[webdriver:106]: Starting webdriver for current session
2024-02-26 15:23:23 DEBUG Process-1[webdriver:122]: Using browser version: 121.0.6167.184
2024-02-26 15:23:23 DEBUG Process-1[webdriver:126]: Loading Southwest Check-In page
2024-02-26 15:23:27 DEBUG Process-1[webdriver:64]: Waiting for valid headers
2024-02-26 15:23:27 DEBUG Process-1[webdriver:155]: Waiting for headers_set to be set
2024-02-26 15:23:27 DEBUG Process-1[webdriver:159]: headers_set set successfully
2024-02-26 15:23:27 DEBUG Process-1[reservation_monitor:84]: Scheduling flight check-ins for 1 reservations
2024-02-26 15:23:27 DEBUG Process-1[checkin_scheduler:76]: Retrieving reservation information
2024-02-26 15:23:41 DEBUG Process-1[utils:39]: Failed to make request after 20 attempts: Forbidden 403
2024-02-26 15:23:41 DEBUG Process-1[utils:42]: Response body: {
"code": 403050700
}
2024-02-26 15:23:41 DEBUG Process-1[checkin_scheduler:82]: Failed to retrieve reservation info. Error: Forbidden 403. Exiting
2024-02-26 15:23:41 DEBUG Process-1[notification_handler:76]: Sending failed reservation retrieval notification...
Failed to retrieve reservation for XXXXXXX XXXXXXXXXXX with confirmation number XXXXXX. Reason: Forbidden 403.
Make sure the reservation information is correct and try again.
2024-02-26 15:23:41 DEBUG Process-1[checkin_scheduler:57]: 0 flights found under current reservation
2024-02-26 15:23:41 DEBUG Process-1[checkin_scheduler:43]: 0 total flights were found
2024-02-26 15:23:41 DEBUG Process-1[checkin_scheduler:100]: 0 new flights found
2024-02-26 15:23:41 DEBUG Process-1[checkin_scheduler:104]: Scheduling 0 flights for check-in
2024-02-26 15:23:41 DEBUG Process-1[checkin_scheduler:116]: 0 flights are currently scheduled. Removing old flights
2024-02-26 15:23:41 DEBUG Process-1[checkin_scheduler:132]: Successfully removed old flights. 0 flights are now scheduled
2024-02-26 15:23:41 DEBUG Process-1[reservation_monitor:71]: No more flights are scheduled for check-in. Exiting...
Additional context
Synology DSM 7.x Docker version 20.10.3, build 55f0773
Btw, from the same household and thus the same source IP, I can hit southwest.com and do a test "CHANGE" flight w/ the flight details and it shows the flight itinerary. I am not sure which method your code is using to retrieve the flight details, though.
Oh, and the record locator in question is currently within the 24 hour checkin period for the departure flight.. and was checked in manually yesterday afternoon. The return flight is on this Wednesday afternoon on the same record locator.
This is the new issue that has been come across in #226 and #201. I will consolidate those issues into this one.
I can also reproduce this issue in Docker too (both using Chromium and Chrome). I believe this is just a change from #201 that Southwest did that now returns the forbidden error instead of a 429 (which makes more sense).
You probably have already seen this; but in case not, see the Outro section on this post regarding methods of bypassing reCaptcha. The reason I bring this up is we may be dealing with an invisible reCaptcha style algorithm.
@hildebrau Hey. If you dont mind sharing, how did you get the webdriver to work in a Synology NAS? I am getting the following error
2024/02/27 00:03:33 | stdout | 2024-02-27 08:01:16 DEBUG Process-1[webdriver:103]: Starting webdriver for current session
[...]
2024/02/27 00:03:33 | stdout | File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 348, in execute
2024/02/27 00:03:33 | stdout | self.error_handler.check_response(response)
2024/02/27 00:03:33 | stdout | File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
2024/02/27 00:03:33 | stdout | selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:9222
2024/02/27 00:03:33 | stdout | raise exception_class(message, screen, stacktrace)
2024/02/27 00:03:33 | stdout | from chrome not reachable
I am guessing it can't find a web browser anywhere on the NAS. How did you get around that? Thanks
how did you get the webdriver to work in a Synology NAS?
@Agiang42, I never saw such an issue. Are you running this code as a docker container? If not, that's probably the issue. I have not tried running the python directly on the Synology because I don't believe all the prerequisites could be installed.
I'm seeing this same issue outside of docker, as well.
Successfully logged in to XXXX XXXX's account
Failed to retrieve reservation for XXXX XXXX with confirmation number XXXXXX.
Reason: Forbidden 403
Failed to retrieve reservation for XXXX XXXX with confirmation number XXXXXX.
Reason: Forbidden 403
Encountered a Too Many Requests error while logging in. Skipping reservation retrieval
I have two flights booked for the end of March, so if there are any logs that would be helpful or anything I can try with my account between now and then, I'll be happy to do so.
I am also having the same issue running outside docker with develop branch. Ubuntu 22.04 python 3.10.12 Chrome 122.0.6261.69 Account type config fails and so does the Reservation type.
Same issue for me as well. Any ideas on a fix? Is there any sense on the logic behind how southwest is determining when to issue the 403?
Any ideas on a fix? Is there any sense on the logic behind how southwest is determining when to issue the 403?
I believe this is the same error as #201, just returning a different status code due to Southwest’s improvements they did. That issue’s thread has many details on what I and others have tried, but there is no known fix to this as of yet. Most people are only getting the error in Docker, so you can try to run it with Python instead for now.
Just wanted to give a heads up that I'm running into the 403 issue in develop running the script, so it's probably not just a Docker issue.
Within Docker, I can still reproduce this issue. However, the requests seem to go through successfully in some parts and fail in others (although they fail at different requests).
I've been analyzing the requests that have gone through, used headed mode for the browser within Docker (using Xvfb in SeleniumBase), and messing around with the initialization flags that are being used to start the browser. However, nothing has been working as of yet.
Very strange how most people are only seeing it within Docker, but a select few are also seeing it outside of Docker. If this issue starts occurring for me outside of Docker, it will be easier to find out what is wrong but I don't currently have anymore ideas to see what the issue is/how to fix it.
I can repro this 403 issue using a linux server running the python command script directly, and also using docker on the same box. However, everything works on a Mac that is my daily driver. Southwest could possibly be doing some fingerprinting and blocking the connection from untrusted or unseen clients that it assumes to be bots?
This makes sense, as I running raw linux and python and seeing the error. Can we spoof the user-agent maybe?
I just tried this from a Ubuntu 22.04 desktop with Python 3.10.12 and Google Chrome 122.0.6261.111 and the master branch, and it seems to work fine along with the fare check... The system that fails to work is Ubuntu 22.04 server with Python 3.10.12 and Google Chrome 122.0.6261.111 running in an LXC virtual machine same internal network as the machine that works.. If you want logs let me know.
Can we spoof the user-agent maybe?
SeleniumBase already changes the user agent to line up with what headed Chrome has. The user agent can also be changed by adding user_agent="<agent>" on line 121 of lib/webdriver.py.
It seems like a lot of people are running into issues specifically with a server, so I will try on an Ubuntu server VM to see if I can reproduce the issue.
I seem to be encountering a combination of this and #96 on my Unraid box.
Running without privileged: true in the docker-compose just hangs until timing out (like in #96 and #147), and adding it allows the webdriver to actually start before hitting the 403
2024-03-06 20:24:52 DEBUG Process-4[reservation_monitor:86]: Scheduling flight check-ins for 1 reservations
2024-03-06 20:24:52 DEBUG Process-4[checkin_scheduler:76]: Retrieving reservation information
2024-03-06 20:25:08 DEBUG Process-4[utils:40]: Failed to make request after 20 attempts: Forbidden 403
2024-03-06 20:25:08 DEBUG Process-4[utils:43]: Response body: {
"code": 403050700
}
2024-03-06 20:25:08 DEBUG Process-4[checkin_scheduler:82]: Failed to retrieve reservation info. Error: Forbidden 403. Exiting
Can we spoof the user-agent maybe?
SeleniumBase already changes the user agent to line up with what headed Chrome has. The user agent can also be changed by adding
user_agent="<agent>"on line 121 oflib/webdriver.py.It seems like a lot of people are running into issues specifically with a server, so I will try on an Ubuntu server VM to see if I can reproduce the issue.
Hey sir! Funny thing, I just setup a droplet/vm (Digital Ocean) to do this, Ubuntu Server 22.04 LTS, getting same error 403, if you are interested I can DM you details how to get into it if you need a setup to troubleshoot that is already built?
It seems like a lot of people are running into issues specifically with a server, so I will try on an Ubuntu server VM to see if I can reproduce the issue.
I can reproduce the issue with an Ubuntu headless server too so that definitely widens the scope of the issue and will make it easier for me to debug (and hopefully fix).
Running without privileged: true in the docker-compose just hangs until timing out (like in https://github.com/jdholtz/auto-southwest-check-in/issues/96 and https://github.com/jdholtz/auto-southwest-check-in/issues/147), and adding it allows the webdriver to actually start before hitting the 403
@drippyer If this is still an issue after the 403 issue is fixed, feel free to make a separate issue for it.
if you are interested I can DM you details how to get into it if you need a setup to troubleshoot that is already built?
@j0nsgh I appreciate the offer. I was able to reproduce it on an Ubuntu server, so now I can troubleshoot this further.
@jdholtz Is there a fix for this? Getting 403s on every reservation I put into the config.json. Same error when just using the command line. Same result in docker and not.
@jdholtz Is there a fix for this?
Currently not. It has been working only on my laptop, which has graphics (desktop, window manager, etc.) so if you aren't already running the script on a device that has graphics you can try that--although it isn't the most convenient. You can also try changing line 118 in lib/webdriver.py from headless=True to headed=True and see if that works.
I'll have more time to work on figuring out this issue after the middle of next week.
FWIW, I was hitting these issues as well. Pulled the latest as of today and ran this through a Docker and it's working.
I've also been monitoring this issue for a fix, but noticed yesterday (without updating versions) that the 403 is gone and was able to retrieve reservations. Perhaps a change was made on Southwest's side.
@jdholtz
I'll have more time to work on figuring out this issue after the middle of next week.
Tried those options, no dice. Brand new install of Ubuntu on a laptop.
I also tried using username and password and got this error: Encountered a Too Many Requests error while logging in. Skipping reservation retrieval.
With this comment: python3 southwest.py CONFIRMATION FIRST LAST, here is the stack trace:
Process Process-1:
Traceback (most recent call last):
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/connection.py", line 198, in _new_conn
sock = connection.create_connection(
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen
response = self._make_request(
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request
raise new_e
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
self._validate_conn(conn)
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
conn.connect()
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/connection.py", line 616, in connect
self.sock = sock = self._new_conn()
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/connection.py", line 205, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0x762eb8369240>: Failed to resolve 'mobile.southwest.com' ([Errno -3] Temporary failure in name resolution)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/schmeed/.local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen
retries = retries.increment(
File "/home/schmeed/.local/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='mobile.southwest.com', port=443): Max retries exceeded with url: /api/mobile-air-booking/v1/mobile-air-booking/page/view-reservation/ZZZCCC?first-name=FIRSTNAME&last-name=LASTNAME (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x762eb8369240>: Failed to resolve 'mobile.southwest.com' ([Errno -3] Temporary failure in name resolution)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/schmeed/auto-southwest-check-in/lib/reservation_monitor.py", line 44, in monitor
self._monitor()
File "/home/schmeed/auto-southwest-check-in/lib/reservation_monitor.py", line 70, in _monitor
self._schedule_reservations([reservation])
File "/home/schmeed/auto-southwest-check-in/lib/reservation_monitor.py", line 88, in _schedule_reservations
self.checkin_scheduler.process_reservations(confirmation_numbers)
File "/home/schmeed/auto-southwest-check-in/lib/checkin_scheduler.py", line 41, in process_reservations
flights.extend(self._get_flights(confirmation_number))
File "/home/schmeed/auto-southwest-check-in/lib/checkin_scheduler.py", line 56, in _get_flights
reservation_info = self._get_reservation_info(confirmation_number)
File "/home/schmeed/auto-southwest-check-in/lib/checkin_scheduler.py", line 77, in _get_reservation_info
response = make_request("GET", site, self.headers, info)
File "/home/schmeed/auto-southwest-check-in/lib/utils.py", line 30, in make_request
response = requests.get(url, headers=headers, params=info)
File "/home/schmeed/.local/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/home/schmeed/.local/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/home/schmeed/.local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/home/schmeed/.local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/home/schmeed/.local/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='mobile.southwest.com', port=443): Max retries exceeded with url: /api/mobile-air-booking/v1/mobile-air-booking/page/view-reservation/ZZZCCC?first-name=FIRSTNAME&last-name=LASTNAME (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x762eb8369240>: Failed to resolve 'mobile.southwest.com' ([Errno -3] Temporary failure in name resolution)"))
schmeed@schmeed-UX31A:~/auto-southwest-check-in$
I've also been monitoring this issue for a fix, but noticed yesterday (without updating versions) that the 403 is gone and was able to retrieve reservations. Perhaps a change was made on Southwest's side.
Unfortunately, I am still getting these issues with Docker and an Ubuntu server. There are random intervals people get where it works, so this may just be one of those times.
I also tried using username and password and got this error: Encountered a Too Many Requests error while logging in. Skipping reservation retrieval.
@schmeed I was getting this error with Chromium, so try Google Chrome if you aren't already using it (uninstall Chromium too so the script doesn't try to use Chromium). As for the traceback, it seems that your DNS might not be resolving correctly due to the NameResolutionError.
I have noticed that southwest is automatically or accelerated blocking many IPs automatically that are known VPS or hosting. I have run the script from both Vultr, DO, and home connection.
Vultr is auto blocked (403) no matter how many IPs I request. DO gets 403/429 with more than 1-2 reservations per IP. My home connection has no problems at all with 4 separate reservations on same flight.
All using Ubuntu 22.04 with headless Google Chrome 122.0.6261.128. Same results with reservations and username/password.
Thank you @jdholtz. I was using the Chromium browser and switching to Chrome resolved the issue. Maybe "Any Chromium based browser" needs revised?
Maybe "Any Chromium based browser" needs revised?
I can change it if many people aren’t running into the issue with Chrome. However, my laptop runs Chromium just fine with this script. On the other hand, I have Chrome on an Ubuntu server and am still reaching the 403.
Is there a way to switch with the docker container?
I have a Dockerfile that uses Chrome that I can send in a few hours. I haven’t been having much success with it, but it would be good to have someone else test it.