Not handling 429's gracefully

[jasonwbarnett]

This may not be directly applicable since I only ran into the error condition after I changed the code in a fork (lowered retry sleep), but it still may make sense to add exception handling for the situation where Southwest rate limits requests and returns a 429. Thinking that adding an additional sleep (i.e. 5 seconds) may make sense.

Response

HTTP/1.1 429 Too Many Requests

Body

{
  "code": 429999999,
  "message": "Error.",
  "messageKey": "ERROR",
  "httpStatusCode": "BAD_REQUEST",
  "requestId": "",
  "infoList": []
}

For reference: https://datatracker.ietf.org/doc/html/rfc6585#section-4

[zxlin]

Can confirm this exact behavior as well - happening from both AWS and residential IP addresses. (Thought it might've been blocking AWS IP, tried residential). This was not happening earlier in the week. Looks like something just changed very recently.

On latest master commit (6738effcc56aa70f008b25af645327431d808079)

[jasonwbarnett]

Yeah I used the bot successfully on Tuesday (August 3rd, 2021). It failed to work yesterday (August 7th, 2021).

[byalextran]

hmm. i wrote a ruby gem that's similar to this. also ran into a 429 error today when things were working a couple months ago. i tried sleeping 30 seconds between calls but still get the error. i think there's something more going on than just rate limiting.

unfortunately, i'm traveling this week and don't have much time to troubleshoot. but hopefully the hive mind will uncover something so we can all fix our code.

[mmoore99]

I have a .Net checkin program that has stopped working as well that is getting the same 429 error described above. I think it may be related to the issue discussed here

[QuiGonLeong]

Same problem here

Sorry! This reservation is not eligible for check in. This flight is currently not ready for check-in. Please try again momentarily. Attempting check-in... Error. Error. Error.

[rbrotz69]

I had the same issue where it just showed about 20 'errors' in a row. I manually checked in and noticed a Covid 19 health acknowledgement screen you need to accept first. I wonder if that is what is screwing up the script?

[dwightmulcahy]

Yeah, noticed it isn't working now... :(

web.1 | 18-Aug 20:39:51-INFO: Attempting check-in...
web.1 | 18-Aug 20:39:51-INFO: Error.
web.1 | 18-Aug 20:39:52-INFO: Error.
web.1 | 18-Aug 20:39:52-INFO: Error.
web.1 | 18-Aug 20:39:53-INFO: Error.
web.1 | 18-Aug 20:39:54-INFO: Error.
web.1 | 18-Aug 20:39:54-INFO: Error.
web.1 | 18-Aug 20:39:55-INFO: Error.

[zxlin]

I had the same issue where it just showed about 20 'errors' in a row. I manually checked in and noticed a Covid 19 health acknowledgement screen you need to accept first. I wonder if that is what is screwing up the script?

it was definitely working this past year when there was a COVID acknowledgement already, I'm inclined to think that is irrelevant.

[jaredhughes2]

I was able to track down the problem, but haven't found a good solution. I'm hoping that others can look at it also and try to find a permanent fix.

The problem is that some headers are missing that weren't previously required. One of them is the user-agent header, which can be hard-coded so it's not a huge deal. But there are several others that are required, and I can't figure out where the values are coming from. The headers all start with ee30zvqlwf, and once I figured out that was the problem, a quick Google search led me to this project: https://github.com/redfern314/southwest-search. There are notes on a temporary fix in the readme file in the project, but it says the workaround might work for a number of days and it only worked for an hour or so for me. It looks like that project was updated with those headers over two years ago, so it looks like it's been a problem for a while (just not with the mobile service until recently).

I'm going to keep looking at it when I have time, but it would be nice if others could try to figure out where those header values are coming from as well.

[zxlin]

This also might be useful reference: https://github.com/Fffrank/southwest-alerts, they're just capturing the headers and using it again. It might be a valid strategy here too.

[schmeed]

I have a .Net checkin program that has stopped working as well that is getting the same 429 error described above. I think it may be related to the issue discussed here

Hello, do you have your .Net code on GH? Willing to share?

[jaredhughes2]

This also might be useful reference: https://github.com/Fffrank/southwest-alerts, they're just capturing the headers and using it again. It might be a valid strategy here too.

I hate the idea of relying on "browser" clicks to get the header values. Ideally somebody will be able to reverse engineer the API and be able to generate the necessary headers each time, but until somebody has time to try to figure that out then this can be used in the mean time. I don't know when I'll have time to do that (might be a couple weeks), but I'll look into it when I can.

[jasonwbarnett]

@jaredhughes2 I've been thinking about this and I was wondering what if we used the Selenium WebDriver to do the browser part automatically with a headless browser? It could be cool, but the additional dependency is annoying.

[rbrotz69]

In the past when the script worked you didn’t have to ‘check in’ but the option to retrieve your boarding pass was there. If you had to ‘check in’ I suspect the script didn’t work.

Southwest has a very interesting way of doing boarding positions. Many times I have gotten low A’s even ahead of people that paid for early bird. SW has an antiquated system and based on cancellations and other factors it’s a bit of a crap shoot.

On Fri, Sep 3, 2021 at 12:09 PM mkitchingh @.***> wrote:

I just had a failed checkin too, but something odd happened, I think. I didn't note for about 15 minutes, and I went to do a manual checkin for my son and I. We got A16 and a17. It made me wonder if the checkin was actually successful, but it the program was unable to detect that. I can't really believe we got 16 and 17 15 minutes after checkin opened.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/pyro2927/SouthwestCheckin/issues/70#issuecomment-912684881, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJREYKM27DSVJA4MAO6JPXLUAD6LJANCNFSM5BXU6U5A .

[rschoen]

I made a temporary workaround, until we figure out something better: https://github.com/rschoen/SouthwestCheckin

Grab the headers from the browser / burp proxy / whatever, and paste them into headers.txt. Then you can run the script as normal. No promises on how often you have to refresh the headers.txt file though. Other sources indicate it's valid for some number of days.

[Jeremy-Harper]

@rschoen I only get a response of access denied. Can you provide more instructions in your fork?

[Jeremy-Harper]

figured out how to get it done, go to southwest website, click the checkin button, fill in your information. Right click in the web browser and say inspect, go to the network tab. Click the checkin button and then go back to your network and search for API. Select southwest which is the correct request and it gets you what you need to paste in @rschoen's workaround.

[Jeremy-Harper]

Can confirm @rschoen script worked this morning

[QuiGonLeong]

@rschoen updates worked correctly. The next step is have it automatically get and update the response headers.

[jasonwbarnett]

I think this could easily be done using a selenium web driver, i.e. https://stackoverflow.com/questions/62262261/how-to-get-request-headers-in-selenium

[jasonwbarnett]

This is some untested code I whipped up that shows the concept.

from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options

REQUEST_URL_WITH_HEADERS = "https://www.southwest.com/api/air-checkin/v1/air-checkin/page/air/check-in/review"

chrome_options = Options()
chrome_options.headless = True
driver = webdriver.Chrome(options=chrome_options)
driver.get("https://www.southwest.com/air/check-in/index.html")

# At this point, fill in check-in form and click submit.

# Search through requests and find the one with the headers
request_with_headers = next((x for x in driver.requests if x.url == REQUEST_URL_WITH_HEADERS), None)
if request_with_headers != None:
    for header in request_with_headers.headers:
        # Pull headers from the first request here.
        print("%s: header", header)
else:
    print("Raise error here")

driver.quit()

[jdholtz]

I've done some testing with a very similar script to this. From my testing, it appears that only the X-Dublriiu-E header needs to be passed (along with the X-Api-Key and X-Channel-Id) to POST the data. The header resets every 5-6 successful tries, but it doesn't seem like it resets after a certain amount of time (although more testing is definitely needed). If we can somehow figure out how that header is generated, we only need to use that one and not all of the headers. *Note: This is all from proxying the Southwest IOS App

[djre4orm]

Thanks to this branch it works as long as you manually pull the headers. I tried letting them rerun for my return flight (3 days later) and the headers were no longer valid, so the time window must be a day or two. I wrote something leveraging what Jason posted above & using browsermob proxy to copy header data. SWA site would not respond well to any selenium driven browser. I had zero luck with firefox. Chrome worked after hex editing the webdriver.exe file https://stackoverflow.com/questions/33225947/can-a-website-detect-when-you-are-using-selenium-with-chromedriver/41220267 The request looks valid. I can copy it as powershell code with the headers & get a valid "Wait till 24hrs" response. However with those automated headers its back to the original error at the time of check-in. Reviewing the headers, the only thing I see missing is "ee30zvqlwf-a0". For some reason with selenium that header doesn't appear. I have tried without the proxy & leaving the automate chrome window open to inspect the headers & still no a0.

[jasonwbarnett]

@djre4orm nice find on the hex replacement to prevent sites from detecting the use of selenium web driver. I'm going to test this with vim on Linux and see if I can get positive results.

[altodd]

I will look into this soon, but if the headers can't be easily reversed I could see the possibility of just making this a multi-docker app and using a dockerized Android Emulator (https://android-developers.googleblog.com/2019/10/continuous-testing-with-new-android.html) to open the app, then steal the headers from that. Or I guess at that point it could just become some automation around all the app functionality, but I'm having trouble imagining anything done that way that won't get broken by a UI change. Potentially we could use adb to broadcast intents to the app. I'm just throwing out some ideas I have as an android developer, if I get some time over Thanksgiving I'll see if I can't experiment with some of those ideas.

[Jeremy-Harper]

Failed to work today, I had updated the file from the network around noon and checkin was at 4pm. It had worked previously.

[djre4orm]

Failed to work today, I had updated the file from the network around noon and checkin was at 4pm. It had worked previously.

Might want to double check your headers, I just had it work this morning with an 11hr setup gap. If you have done anything funny with selenium or browser automation pulling headers can get messed up. My recommendation is use incognito window in that case.

Here are my headers from a working run :authority: :method: :path: :scheme: accept: accept-encoding: accept-language: authorization: content-length: content-type: cookie: ee30zvqlwf-a: ee30zvqlwf-a0: ee30zvqlwf-b: ee30zvqlwf-c: ee30zvqlwf-d: ee30zvqlwf-f: ee30zvqlwf-z: origin: referer: sec-ch-ua: sec-ch-ua-mobile: sec-ch-ua-platform: sec-fetch-dest: sec-fetch-mode: sec-fetch-site: user-agent: x-api-idtoken: x-api-key: x-app-id: x-channel-id: x-kl-ajax-request: x-user-experience-id:

[byalextran]

i appreciate all the eyes on this.

if i understand the problem, southwest added some new request headers that need to be used when checking in. since we don't know how those headers are created, they have to be scraped. but they seem to change regularly (under four hours?) so that new request headers are needed after a certain period.

if all of that is true, i think we can take @jasonwbarnett's code and programmatically get the request headers before each auto check-in attempt.

here's an extension of his code. if you pass in the confirmation number, first name, and last name, the code will output the necessary headers into a JSON file (request_headers.json) which could then be read later when needed.

https://gist.github.com/byalextran/e38cf6321eb78a3a1f79cd282ab69314

i don't have an active confirmation number to test the solution (i will wednesday) but throwing the idea out now for dialogue in the mean time.

if someone has a confirmation number they'd be willing to let me test with beforehand, let me know via my contact form. https://www.alextran.org/contact/

[flyize]

You can always book a flight, try to check in, and then cancel for a refund. They are required by law to give you a full refund within 24 hours of buying the ticket.

[MyCodeIsSchmidt]

You can always book a flight, try to check in, and then cancel for a refund. They are required by law to give you a full refund within 24 hours of buying the ticket.

This does not apply to flights booked less than a week before the flight date.