visa_rescheduler
visa_rescheduler copied to clipboard
uxDaniel
Unable to continue with date searching
I'm having the same issue, it seems that the driver is unable to get the json file from DATE_URL. Stuck tho
Hi, same here, it seems that DATE_URL query is not retrieving the dates even getting the 304 from the request in the web site.
I refreshed the page on my own and checked the request headers. It seems the get URL is not the main reason for the problem. A header problem has caused the 404 Error. So, I checked the new request header and compared it with the manual get request, and these were the differences between the two requests. My Manual Get Request:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Cache-Control: max-age=0
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
Original Request:
Accept: application/json, text/javascript, */*; q=0.01
If-None-Match: W/"8671577069a0606a2686d38cfeae33c5"
Referer: https://ais.usvisa-info.com/en-am/niv/schedule/46352408/appointment
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
X-CSRF-Token: 1Dc0N4Eel+gC1QQq7+h+ouqGyNbI5CDCwBj+CUer8EImwasZVUJ/KgJNDCYdEUSeD2VD2pJ5yds2w4MWsENisg==
X-Requested-With: XMLHttpRequest
I think it's the Referer. We should set the Referer into the header before requesting. I didn't check it yet. However, I believe this is the cause. CSRF can be the next reason.
The full view:
Hi, Do you have any suggestions on how to resolve this issue, specifically regarding how we can add headers when calling "driver.get(DATE_URL)"?
This may solve the problem.
return driver.execute_script("var req = new XMLHttpRequest();req.open('GET', '" + str(url) + "', false);req.setRequestHeader('Accept', 'application/json, text/javascript, */*; q=0.01');req.setRequestHeader('X-Requested-With', 'XMLHttpRequest'); req.setRequestHeader('Cookie', '_yatri_session=" + session + "'); req.send(null);return req.responseText;")
Hi @jaimbox,
Can you please help where to input this?
With Selenium in Python you can run javascript, put it in the place where you need to get the available dates.
Hi I was able to get the dates by adding the following code:
def get_date(self):
self.driver.get(DATE_URL)
if not self.is_logged_in():
self.login()
return self.get_date()
else:
headers = {
"accept": "application/json",
"User-Agent": self.driver.execute_script("return navigator.userAgent;"),
"Referer": f"https://ais.usvisa-info.com/{COUNTRY_CODE}/niv/schedule/{SCHEDULE_ID}/appointmenmt",
"Cookie": "_yatri_session=" + self.driver.get_cookie("_yatri_session")["value"],
'X-Requested-With':'XMLHttpRequest',
}
#print(headers)
r = requests.get(DATE_URL, headers=headers)
#print(r.status_code, r.json())
#content = self.driver.find_element(By.TAG_NAME, 'pre').text
#date = json.loads(content)
date = r.json()
return date
Now I'm trying the same for TIME_URL, but no luck.
Output from the execution:
`INFO:main - ---START--- : 2023-04-05 14:05:44.054362
INFO:login - Login start...
INFO:login - click bounce
INFO:do_login_action - input email
INFO:do_login_action - input pwd
INFO:do_login_action - click privacy
INFO:do_login_action - commit
INFO:do_login_action - login successful!
Dates inside try [{'date': '2024-06-06', 'business_day': True}, {'date': '2024-06-07', 'business_day': True}, {'date': '2025-05-12', 'business_day': True}, {'date': '2025-05-13', 'business_day': True}, {'date': '2025-05-14', 'business_day': True}]
INFO:print_dates - Available dates:
INFO:print_dates - 2024-06-06 business_day: True
INFO:print_dates - 2024-06-07 business_day: True
INFO:print_dates - 2025-05-12 business_day: True
INFO:print_dates - 2025-05-13 business_day: True
INFO:print_dates - 2025-05-14 business_day: True
INFO:get_available_date - Checking for an earlier date:
INFO:is_earlier - Is 2025-04-02 00:00:00 > 2024-06-06 00:00:00: True
2024-06-06
200 {'available_times': [], 'business_times': []}
{'available_times': [], 'business_times': []}
INFO:main - No TIMES available on FACILITY`
When inspecting the operation in chrome get retrieves the correct information.
Hi I was able to get the dates by adding the following code:
def get_date(self): self.driver.get(DATE_URL) if not self.is_logged_in(): self.login() return self.get_date() else: headers = { "accept": "application/json", "User-Agent": self.driver.execute_script("return navigator.userAgent;"), "Referer": f"https://ais.usvisa-info.com/{COUNTRY_CODE}/niv/schedule/{SCHEDULE_ID}/appointmenmt", "Cookie": "_yatri_session=" + self.driver.get_cookie("_yatri_session")["value"], 'X-Requested-With':'XMLHttpRequest', } #print(headers) r = requests.get(DATE_URL, headers=headers) #print(r.status_code, r.json()) #content = self.driver.find_element(By.TAG_NAME, 'pre').text #date = json.loads(content) date = r.json() return dateNow I'm trying the same for TIME_URL, but no luck.
Output from the execution:
`INFO:main - ---START--- : 2023-04-05 14:05:44.054362 INFO:login - Login start... INFO:login - click bounce INFO:do_login_action - input email INFO:do_login_action - input pwd INFO:do_login_action - click privacy INFO:do_login_action - commit INFO:do_login_action - login successful! Dates inside try [{'date': '2024-06-06', 'business_day': True}, {'date': '2024-06-07', 'business_day': True}, {'date': '2025-05-12', 'business_day': True}, {'date': '2025-05-13', 'business_day': True}, {'date': '2025-05-14', 'business_day': True}] INFO:print_dates - Available dates: INFO:print_dates - 2024-06-06 business_day: True INFO:print_dates - 2024-06-07 business_day: True INFO:print_dates - 2025-05-12 business_day: True INFO:print_dates - 2025-05-13 business_day: True INFO:print_dates - 2025-05-14 business_day: True INFO:get_available_date - Checking for an earlier date: INFO:is_earlier - Is 2025-04-02 00:00:00 > 2024-06-06 00:00:00: True 2024-06-06 200 {'available_times': [], 'business_times': []} {'available_times': [], 'business_times': []} INFO:main - No TIMES available on FACILITY`
The following works for me, with the @jaimbox code, when loading date you need to load all the dates in the appointment site
driver.get(APPOINTMENT_URL)
session = driver.get_cookie("_yatri_session")["value"]
NEW_GET = driver.execute_script("var req = new XMLHttpRequest();req.open('GET', '" + str(DATE_URL) + "', false);req.setRequestHeader('Accept', 'application/json, text/javascript, */*; q=0.01');req.setRequestHeader('X-Requested-With', 'XMLHttpRequest'); req.setRequestHeader('Cookie', '_yatri_session=" + session + "'); req.send(null);return req.responseText;")
For times, you just have to do the same in get_time fonction. For example
time_url = TIME_URL % date
session = driver.get_cookie("_yatri_session")["value"]
NEW_TIME = driver.execute_script("var req = new XMLHttpRequest();req.open('GET', '" + str(time_url) + "', false);req.setRequestHeader('Accept', 'application/json, text/javascript, */*; q=0.01');req.setRequestHeader('X-Requested-With', 'XMLHttpRequest'); req.setRequestHeader('Cookie', '_yatri_session=" + session + "'); req.send(null);return req.responseText;")
Hi I was able to get the dates by adding the following code:
def get_date(self): self.driver.get(DATE_URL) if not self.is_logged_in(): self.login() return self.get_date() else: headers = { "accept": "application/json", "User-Agent": self.driver.execute_script("return navigator.userAgent;"), "Referer": f"https://ais.usvisa-info.com/{COUNTRY_CODE}/niv/schedule/{SCHEDULE_ID}/appointmenmt", "Cookie": "_yatri_session=" + self.driver.get_cookie("_yatri_session")["value"], 'X-Requested-With':'XMLHttpRequest', } #print(headers) r = requests.get(DATE_URL, headers=headers) #print(r.status_code, r.json()) #content = self.driver.find_element(By.TAG_NAME, 'pre').text #date = json.loads(content) date = r.json() return dateNow I'm trying the same for TIME_URL, but no luck. Output from the execution:
`INFO:main - ---START--- : 2023-04-05 14:05:44.054362 INFO:login - Login start... INFO:login - click bounce INFO:do_login_action - input email INFO:do_login_action - input pwd INFO:do_login_action - click privacy INFO:do_login_action - commit INFO:do_login_action - login successful! Dates inside try [{'date': '2024-06-06', 'business_day': True}, {'date': '2024-06-07', 'business_day': True}, {'date': '2025-05-12', 'business_day': True}, {'date': '2025-05-13', 'business_day': True}, {'date': '2025-05-14', 'business_day': True}] INFO:print_dates - Available dates: INFO:print_dates - 2024-06-06 business_day: True INFO:print_dates - 2024-06-07 business_day: True INFO:print_dates - 2025-05-12 business_day: True INFO:print_dates - 2025-05-13 business_day: True INFO:print_dates - 2025-05-14 business_day: True INFO:get_available_date - Checking for an earlier date: INFO:is_earlier - Is 2025-04-02 00:00:00 > 2024-06-06 00:00:00: True 2024-06-06 200 {'available_times': [], 'business_times': []} {'available_times': [], 'business_times': []} INFO:main - No TIMES available on FACILITY`The following works for me, with the @jaimbox code, when loading date you need to load all the dates in the appointment site
driver.get(APPOINTMENT_URL) session = driver.get_cookie("_yatri_session")["value"] NEW_GET = driver.execute_script("var req = new XMLHttpRequest();req.open('GET', '" + str(DATE_URL) + "', false);req.setRequestHeader('Accept', 'application/json, text/javascript, */*; q=0.01');req.setRequestHeader('X-Requested-With', 'XMLHttpRequest'); req.setRequestHeader('Cookie', '_yatri_session=" + session + "'); req.send(null);return req.responseText;")For times, you just have to do the same in get_time fonction. For example
time_url = TIME_URL % date session = driver.get_cookie("_yatri_session")["value"] NEW_TIME = driver.execute_script("var req = new XMLHttpRequest();req.open('GET', '" + str(time_url) + "', false);req.setRequestHeader('Accept', 'application/json, text/javascript, */*; q=0.01');req.setRequestHeader('X-Requested-With', 'XMLHttpRequest'); req.setRequestHeader('Cookie', '_yatri_session=" + session + "'); req.send(null);return req.responseText;")
Hi thanks for the answer. I'm not able to use cookie in the execute_script. This is what I got:
DevTools listening on ws://127.0.0.1:56632/devtools/browser/95491e8f-fdd6-4296-9213-f47668005cb0
INFO:main - ---START--- : 2023-04-05 18:00:48.825406
INFO:login - Login start...
INFO:login - click bounce
INFO:do_login_action - input email
INFO:do_login_action - input pwd
INFO:do_login_action - click privacy
INFO:do_login_action - commit
INFO:do_login_action - login successful!
[0405/180124.270:INFO:CONSOLE(3)] "Refused to set unsafe header "Cookie"", source: (3)
@Camilo2015 could you please upload the latest version with updates entire page?
@Orastikroat I would like to, but now I'm struggling with the site IP blocking. After a couple of uploads, the site doesn't load and I don't know if it's really the IP. I don't know if you have the same issue.
Here is the partial script that I integrated all knowledge shared from this conversation. All credits to @jaimbox and @Camilo2015. It's pretty messy, but it works for me, for both get available dates and rescheduling. In specific, the functions that I made changes are get_date and get_time
import time
import json
import random
import platform
import configparser
from datetime import datetime
import requests
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait as Wait
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
from sendgrid import SendGridAPIClient
from sendgrid.helpers.mail import Mail
from typing import List, Tuple
config = configparser.ConfigParser()
config.read('config.ini')
USERNAME = config['USVISA']['USERNAME']
PASSWORD = config['USVISA']['PASSWORD']
SCHEDULE_ID = config['USVISA']['SCHEDULE_ID']
MY_SCHEDULE_DATE = config['USVISA']['MY_SCHEDULE_DATE']
COUNTRY_CODE = config['USVISA']['COUNTRY_CODE']
FACILITY_ID = config['USVISA']['FACILITY_ID']
SENDGRID_API_KEY = config['SENDGRID']['SENDGRID_API_KEY']
PUSH_TOKEN = config['PUSHOVER']['PUSH_TOKEN']
PUSH_USER = config['PUSHOVER']['PUSH_USER']
LOCAL_USE = config['CHROMEDRIVER'].getboolean('LOCAL_USE')
HUB_ADDRESS = config['CHROMEDRIVER']['HUB_ADDRESS']
REGEX_CONTINUE = "//a[contains(text(),'Continue')]"
# def MY_CONDITION(month, day): return int(month) == 11 and int(day) >= 5
def MY_CONDITION(month, day): return True # No custom condition wanted for the new scheduled date
STEP_TIME = 0.5 # time between steps (interactions with forms): 0.5 seconds
RETRY_TIME = 60*10 # wait time between retries/checks for available dates: 10 minutes
EXCEPTION_TIME = 60*30 # wait time when an exception occurs: 30 minutes
COOLDOWN_TIME = 60*60 # wait time when temporary banned (empty list): 60 minutes
DATE_URL = f"https://ais.usvisa-info.com/{COUNTRY_CODE}/niv/schedule/{SCHEDULE_ID}/appointment/days/{FACILITY_ID}.json?appointments[expedite]=false"
TIME_URL = f"https://ais.usvisa-info.com/{COUNTRY_CODE}/niv/schedule/{SCHEDULE_ID}/appointment/times/{FACILITY_ID}.json?date=%s&appointments[expedite]=false"
APPOINTMENT_URL = f"https://ais.usvisa-info.com/{COUNTRY_CODE}/niv/schedule/{SCHEDULE_ID}/appointment"
EXIT = False
def send_notification(msg):
print(f"Sending notification: {msg}")
if SENDGRID_API_KEY:
message = Mail(
from_email=USERNAME,
to_emails=USERNAME,
subject=msg,
html_content=msg)
try:
sg = SendGridAPIClient(SENDGRID_API_KEY)
response = sg.send(message)
print(response.status_code)
print(response.body)
print(response.headers)
except Exception as e:
print(e.message)
if PUSH_TOKEN:
url = "https://api.pushover.net/1/messages.json"
data = {
"token": PUSH_TOKEN,
"user": PUSH_USER,
"message": msg
}
requests.post(url, data)
def get_driver():
if LOCAL_USE:
dr = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
else:
dr = webdriver.Remote(
command_executor=HUB_ADDRESS, options=webdriver.ChromeOptions()
)
return dr
driver = get_driver()
def login():
# Bypass reCAPTCHA
driver.get(f"https://ais.usvisa-info.com/{COUNTRY_CODE}/niv")
time.sleep(STEP_TIME)
a = driver.find_element(By.XPATH, '//a[@class="down-arrow bounce"]')
a.click()
time.sleep(STEP_TIME)
print("Login start...")
href = driver.find_element(
By.XPATH, '//*[@id="header"]/nav/div[1]/div[1]/div[2]/div[1]/ul/li[3]/a'
)
href.click()
time.sleep(STEP_TIME)
Wait(driver, 60).until(EC.presence_of_element_located((By.NAME, "commit")))
print("\tclick bounce")
a = driver.find_element(By.XPATH, '//a[@class="down-arrow bounce"]')
a.click()
time.sleep(STEP_TIME)
do_login_action()
def do_login_action():
print("\tinput email")
user = driver.find_element(By.ID, "user_email")
user.send_keys(USERNAME)
time.sleep(random.randint(1, 3))
print("\tinput pwd")
pw = driver.find_element(By.ID, "user_password")
pw.send_keys(PASSWORD)
time.sleep(random.randint(1, 3))
print("\tclick privacy")
box = driver.find_element(By.CLASS_NAME, "icheckbox")
box.click()
time.sleep(random.randint(1, 3))
print("\tcommit")
btn = driver.find_element(By.NAME, "commit")
btn.click()
time.sleep(random.randint(1, 3))
Wait(driver, 60).until(EC.presence_of_element_located((By.XPATH, REGEX_CONTINUE)))
print("\tlogin successful!")
def get_date():
# driver.get(DATE_URL)
driver.get(APPOINTMENT_URL)
session = driver.get_cookie("_yatri_session")["value"]
NEW_GET = driver.execute_script(
"var req = new XMLHttpRequest();req.open('GET', '"
+ str(DATE_URL)
+ "', false);req.setRequestHeader('Accept', 'application/json, text/javascript, */*; q=0.01');req.setRequestHeader('X-Requested-With', 'XMLHttpRequest'); req.setRequestHeader('Cookie', '_yatri_session="
+ session
+ "'); req.send(null);return req.responseText;"
)
return json.loads(NEW_GET)
def get_time(date):
time_url = TIME_URL % date
session = driver.get_cookie("_yatri_session")["value"]
content = driver.execute_script(
"var req = new XMLHttpRequest();req.open('GET', '"
+ str(time_url)
+ "', false);req.setRequestHeader('Accept', 'application/json, text/javascript, */*; q=0.01');req.setRequestHeader('X-Requested-With', 'XMLHttpRequest'); req.setRequestHeader('Cookie', '_yatri_session="
+ session
+ "'); req.send(null);return req.responseText;"
)
data = json.loads(content)
time = data.get("available_times")[-1]
print(f"Got time successfully! {date} {time}")
return time
def reschedule(date):
global EXIT
print(f"Starting Reschedule ({date})")
time = get_time(date)
driver.get(APPOINTMENT_URL)
data = {
"utf8": driver.find_element(by=By.NAME, value="utf8").get_attribute("value"),
"authenticity_token": driver.find_element(
by=By.NAME, value="authenticity_token"
).get_attribute("value"),
"confirmed_limit_message": driver.find_element(
by=By.NAME, value="confirmed_limit_message"
).get_attribute("value"),
"use_consulate_appointment_capacity": driver.find_element(
by=By.NAME, value="use_consulate_appointment_capacity"
).get_attribute("value"),
"appointments[consulate_appointment][facility_id]": FACILITY_ID,
"appointments[consulate_appointment][date]": date,
"appointments[consulate_appointment][time]": time,
}
headers = {
"User-Agent": driver.execute_script("return navigator.userAgent;"),
"Referer": APPOINTMENT_URL,
"Cookie": "_yatri_session=" + driver.get_cookie("_yatri_session")["value"],
}
r = requests.post(APPOINTMENT_URL, headers=headers, data=data)
if r.text.find("You have successfully scheduled your visa appointment") != -1:
msg = f"Rescheduled Successfully! {date} {time}"
send_notification(msg)
EXIT = True
else:
msg = f"Reschedule Failed. {date} {time}: {r.text} \n\n{r.status_code}"
send_notification(msg)
def is_logged_in():
content = driver.page_source
if content.find("error") != -1:
return False
return True
def print_dates(dates):
print("Available dates:")
print(dates)
for d in dates:
print("%s \t business_day: %s" % (d.get("date"), d.get("business_day")))
print()
last_seen = None
def get_available_date(dates):
global last_seen
def is_earlier(date):
my_date = datetime.strptime(MY_SCHEDULE_DATE, "%Y-%m-%d")
new_date = datetime.strptime(date, "%Y-%m-%d")
result = my_date > new_date
print(f"Is {my_date} > {new_date}:\t{result}")
return result
print("Checking for an earlier date:")
for d in dates:
date = d.get("date")
if is_earlier(date) and date != last_seen:
_, month, day = date.split("-")
if MY_CONDITION(month, day):
last_seen = date
return date
def push_notification(dates):
msg = "date: "
for d in dates:
msg = msg + d.get("date") + "; "
send_notification(msg)
if __name__ == "__main__":
login()
retry_count = 0
while 1:
if retry_count > 6:
break
try:
print("------------------")
print(datetime.today())
print(f"Retry count: {retry_count}")
print()
dates = get_date()[:5]
if not dates:
msg = "List is empty"
send_notification(msg)
# EXIT = True
print_dates(dates)
date = get_available_date(dates)
print()
print(f"New date: {date}")
if date:
reschedule(date)
push_notification(dates)
if EXIT:
print("------------------exit")
break
if not dates:
msg = "List is empty"
send_notification(msg)
# EXIT = True
time.sleep(COOLDOWN_TIME)
else:
time.sleep(RETRY_TIME)
except Exception as e:
retry_count += 1
time.sleep(EXCEPTION_TIME)
from traceback import format_exc
send_notification("Program crashed!" + str(format_exc()))
if not EXIT:
send_notification("HELP! Crashed.")
def get_driver(): if LOCAL_USE: dr = webdriver.Chrome(service=Service(ChromeDriverManager().install())) else: dr = webdriver.Remote(command_executor=HUB_ADDRESS, options=webdriver.ChromeOptions()) return dr
The above needs to be inserted before
driver = get_driver()
@AdriandLiu you're a genius!!! thank you so much brother much appreciate it! it works after the driver amendment
def get_driver(): if LOCAL_USE: dr = webdriver.Chrome(service=Service(ChromeDriverManager().install())) else: dr = webdriver.Remote(command_executor=HUB_ADDRESS, options=webdriver.ChromeOptions()) return dr
The above needs to be inserted before
driver = get_driver()
Yah, I was aware of that, it was a draft. Thanks for pointing it out, I have added the missing functions to the comment.
@AdriandLiu will it work for group ?
I'm not sure, feel free to test it yourself and let us know the results.
@AdriandLiu I am sorry to to inform you that I do not have any programming background and I was looking to expedite my family appointment prior to the current date. I was searching online and ended up in this discussion.
@AdriandLiu thanks for your help I've been doing some changes in order for the script to work, but I'm stuck on MY_CONDITION.
`Traceback (most recent call last):
File "visa_fix.py", line 250, in <module>
date = get_available_date(dates)
File "visa_fix.py", line 220, in get_available_date
if MY_CONDITION(month, day):
NameError: name 'MY_CONDITION' is not defined
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "visa_fix.py", line 271, in <module>
time.sleep(EXCEPTION_TIME)
NameError: name 'EXCEPTION_TIME' is not defined`
@sebaz9130
My bad, I have added the missing variables/functions there, please check the latest updates of https://github.com/uxDaniel/visa_rescheduler/issues/28#issuecomment-1499151303
@AdriandLiu I am sorry to to inform you that I do not have any programming background and I was looking to expedite my family appointment prior to the current date. I was searching online and ended up in this discussion.
Sorry mine is individual appointment request, afraid my inputs are not helpful here.
Happily, it now works, but after a few searches, it will be banned. I'm not sure if this is an IP level ban.
@AdriandLiu Thanks!
Don't we need to check for login in get_date():
if not is_logged_in():
login()
return get_date()
@Orastikroat my apologies for not putting the code, it is a mess for now in my PC, thanks @AdriandLiu for sharing the clean code. As @JesseWest7 I'm being also banned, some ideas about it? Even if we change of IP, even for just 4 searches, we'll be banned, so impossble to keep a VPN changing constantly.