overleaf-sync
overleaf-sync copied to clipboard
moritzgloeckl
'NoneType' object has no attribute 'get'
when trying to run ols list i get the following response:
💥 Querying all projects
Error: Querying all projects failed. Please try again.
with verbose logging i get this traceback additionally
Traceback (most recent call last):
File "/home/emil/.local/lib/python3.10/site-packages/olsync/olsync.py", line 348, in execute_action
success = action()
File "/home/emil/.local/lib/python3.10/site-packages/olsync/olsync.py", line 175, in <lambda>
lambda: overleaf_client.get_project(project_name),
File "/home/emil/.local/lib/python3.10/site-packages/olsync/olclient.py", line 104, in get_project
BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': 'ol-projects'}).get('content'))
AttributeError: 'NoneType' object has no attribute 'get'
i reinstalled all python dependencies and qt6-webengine and the problem still persists
ols used to work for me before, but recently I am facing the same issue.
ols used to work for me before, but recently I am facing the same issue.
me too
when trying to run
ols listi get the following response:💥 Querying all projects Error: Querying all projects failed. Please try again.with verbose logging i get this traceback additionally
Traceback (most recent call last): File "/home/emil/.local/lib/python3.10/site-packages/olsync/olsync.py", line 348, in execute_action success = action() File "/home/emil/.local/lib/python3.10/site-packages/olsync/olsync.py", line 175, in <lambda> lambda: overleaf_client.get_project(project_name), File "/home/emil/.local/lib/python3.10/site-packages/olsync/olclient.py", line 104, in get_project BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': 'ol-projects'}).get('content')) AttributeError: 'NoneType' object has no attribute 'get'i reinstalled all python dependencies and qt6-webengine and the problem still persists I seem to solve this it's because the change of overleaf I think. There's no ol-projects anymore, but ol-prefetchedProjectsBlob copy these python code and cover the original olclient.py seems to have some format problem, sorry I dont know how to write md document
'''
import requests as reqs from bs4 import BeautifulSoup import json import uuid from socketIO_client import SocketIO import time
LOGIN_URL = "https://www.overleaf.com/login" PROJECT_URL = "https://www.overleaf.com/project" # The dashboard URL
DOWNLOAD_URL = "https://www.overleaf.com/project/{}/download/zip" UPLOAD_URL = "https://www.overleaf.com/project/{}/upload" # The URL to upload files FOLDER_URL = "https://www.overleaf.com/project/{}/folder" # The URL to create folders DELETE_URL = "https://www.overleaf.com/project/{}/doc/{}" # The URL to delete files COMPILE_URL = "https://www.overleaf.com/project/{}/compile?enable_pdf_caching=true" # The URL to compile the project BASE_URL = "https://www.overleaf.com" # The Overleaf Base URL PATH_SEP = "/" # Use hardcoded path separator for both windows and posix system class OverleafClient(object): @staticmethod def filter_projects(json_content, more_attrs=None): more_attrs = more_attrs or {} for p in json_content: if not p.get("archived") and not p.get("trashed"): if all(p.get(k) == v for k, v in more_attrs.items()): yield p
def __init__(self, cookie=None, csrf=None):
self._cookie = cookie # Store the cookie for authenticated requests
self._csrf = csrf # Store the CSRF token since it is needed for some requests
def login(self, username, password):
get_login = reqs.get(LOGIN_URL)
self._csrf = BeautifulSoup(get_login.content, 'html.parser').find(
'input', {'name': '_csrf'}).get('value')
login_json = {
"_csrf": self._csrf,
"email": username,
"password": password
}
post_login = reqs.post(LOGIN_URL, json=login_json,
cookies=get_login.cookies)
# On a successful authentication the Overleaf API returns a new authenticated cookie.
# If the cookie is different than the cookie of the GET request the authentication was successful
if post_login.status_code == 200 and get_login.cookies["overleaf_session2"] != post_login.cookies[
"overleaf_session2"]:
self._cookie = post_login.cookies
# Enrich cookie with GCLB cookie from GET request above
self._cookie['GCLB'] = get_login.cookies['GCLB']
# CSRF changes after making the login request, new CSRF token will be on the projects page
projects_page = reqs.get(PROJECT_URL, cookies=self._cookie)
self._csrf = BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': 'ol-csrfToken'}) \
.get('content')
return {"cookie": self._cookie, "csrf": self._csrf}
def all_projects(self):
projects_page = reqs.get(PROJECT_URL, cookies=self._cookie)
# print(BeautifulSoup(projects_page.content, 'html.parser'))
# json_content = json.loads(
# BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': 'ol-projects'}).get('content'))
testname='ol-prefetchedProjectsBlob'
# print(BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': testname}).get('content'))
# print(type(BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': testname}).get('content')))
json_content = json.loads(
(BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': testname}).get('content')))
json_content=json_content.get('projects')
# print(((json_content.get('projects')[0])))
# print(list(OverleafClient.filter_projects(json_content)))
return list(OverleafClient.filter_projects(json_content))
# test2=OverleafClient.filter_projects((json_content.get('projects')[0]))
# return (list(test2))
def get_project(self, project_name):
projects_page = reqs.get(PROJECT_URL, cookies=self._cookie)
json_content = json.loads(
BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': 'ol-prefetchedProjectsBlob'}).get('content'))
return next(OverleafClient.filter_projects(json_content, {"name": project_name}), None)
def download_project(self, project_id):
r = reqs.get(DOWNLOAD_URL.format(project_id),
stream=True, cookies=self._cookie)
return r.content
def create_folder(self, project_id, parent_folder_id, folder_name):
params = {
"parent_folder_id": parent_folder_id,
"name": folder_name
}
headers = {
"X-Csrf-Token": self._csrf
}
r = reqs.post(FOLDER_URL.format(project_id),
cookies=self._cookie, headers=headers, json=params)
if r.ok:
return json.loads(r.content)
elif r.status_code == str(400):
# Folder already exists
return
else:
raise reqs.HTTPError()
def get_project_infos(self, project_id):
project_infos = None
# Callback function for the joinProject emitter
def set_project_infos(a, project_infos_dict, c, d):
# Set project_infos variable in outer scope
nonlocal project_infos
project_infos = project_infos_dict
# Convert cookie from CookieJar to string
cookie = "GCLB={}; overleaf_session2={}" \
.format(
self._cookie["GCLB"],
self._cookie["overleaf_session2"]
)
# Connect to Overleaf Socket.IO, send a time parameter and the cookies
socket_io = SocketIO(
BASE_URL,
params={'t': int(time.time())},
headers={'Cookie': cookie}
)
# Wait until we connect to the socket
socket_io.on('connect', lambda: None)
socket_io.wait_for_callbacks()
# Send the joinProject event and receive the project infos
socket_io.emit('joinProject', {'project_id': project_id}, set_project_infos)
socket_io.wait_for_callbacks()
# Disconnect from the socket if still connected
if socket_io.connected:
socket_io.disconnect()
return project_infos
def upload_file(self, project_id, project_infos, file_name, file_size, file):
# Set the folder_id to the id of the root folder
folder_id = project_infos['rootFolder'][0]['_id']
# The file name contains path separators, check folders
if PATH_SEP in file_name:
local_folders = file_name.split(PATH_SEP)[:-1] # Remove last item since this is the file name
current_overleaf_folder = project_infos['rootFolder'][0]['folders'] # Set the current remote folder
for local_folder in local_folders:
exists_on_remote = False
for remote_folder in current_overleaf_folder:
# Check if the folder exists on remote, continue with the new folder structure
if local_folder.lower() == remote_folder['name'].lower():
exists_on_remote = True
folder_id = remote_folder['_id']
current_overleaf_folder = remote_folder['folders']
break
# Create the folder if it doesn't exist
if not exists_on_remote:
new_folder = self.create_folder(project_id, folder_id, local_folder)
current_overleaf_folder.append(new_folder)
folder_id = new_folder['_id']
current_overleaf_folder = new_folder['folders']
params = {
"folder_id": folder_id,
"_csrf": self._csrf,
"qquuid": str(uuid.uuid4()),
"qqfilename": file_name,
"qqtotalfilesize": file_size,
}
files = {
"qqfile": file
}
# Upload the file to the predefined folder
r = reqs.post(UPLOAD_URL.format(project_id), cookies=self._cookie, params=params, files=files)
return r.status_code == str(200) and json.loads(r.content)["success"]
def delete_file(self, project_id, project_infos, file_name):
file = None
# The file name contains path separators, check folders
if PATH_SEP in file_name:
local_folders = file_name.split(PATH_SEP)[:-1] # Remove last item since this is the file name
current_overleaf_folder = project_infos['rootFolder'][0]['folders'] # Set the current remote folder
for local_folder in local_folders:
for remote_folder in current_overleaf_folder:
if local_folder.lower() == remote_folder['name'].lower():
file = next((v for v in remote_folder['docs'] if v['name'] == file_name.split(PATH_SEP)[-1]),
None)
current_overleaf_folder = remote_folder['folders']
break
# File is in root folder
else:
file = next((v for v in project_infos['rootFolder'][0]['docs'] if v['name'] == file_name), None)
# File not found!
if file is None:
return False
headers = {
"X-Csrf-Token": self._csrf
}
r = reqs.delete(DELETE_URL.format(project_id, file['_id']), cookies=self._cookie, headers=headers, json={})
return r.status_code == str(204)
def download_pdf(self, project_id):
headers = {
"X-Csrf-Token": self._csrf
}
body = {
"check": "silent",
"draft": False,
"incrementalCompilesEnabled": True,
"rootDoc_id": "",
"stopOnFirstError": False
}
r = reqs.post(COMPILE_URL.format(project_id), cookies=self._cookie, headers=headers, json=body)
if not r.ok:
raise reqs.HTTPError()
compile_result = json.loads(r.content)
if compile_result["status"] != "success":
raise reqs.HTTPError()
pdf_file = next(v for v in compile_result['outputFiles'] if v['type'] == 'pdf')
download_req = reqs.get(BASE_URL + pdf_file['url'], cookies=self._cookie, headers=headers)
if download_req.ok:
return pdf_file['path'], download_req.content
return None
'''
ols used to work for me before, but recently I am facing the same issue.
me too
me too
Change olclient.py line 92 to
BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': 'ol-prefetchedProjectsBlob'}).get('content'))['projects'] will fix this issue. This is due to the overleaf webpage is changed, and the original code cannot find projects anymore. Maybe I will make a pull request later.
Change olclient.py line 92 to
BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': 'ol-prefetchedProjectsBlob'}).get('content'))['projects']will fix this issue. This is due to the overleaf webpage is changed, and the original code cannot find projects anymore. Maybe I will make a pull request later.
It works. That's cool!
Change olclient.py line 92 to
BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': 'ol-prefetchedProjectsBlob'}).get('content'))['projects']will fix this issue. This is due to the overleaf webpage is changed, and the original code cannot find projects anymore. Maybe I will make a pull request later.
Thank you for your response. The command ols list executed successfully, however, the command 'ols download -n "test"' did not. It returned the following error message:
$ ols download -n test
💥 Querying project
💥 Downloading project's PDF
Error: Downloading project's PDF failed. Please try again.
I would appreciate it if you could look into this issue further. Thanks in advance.
Change olclient.py line 92 to
BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': 'ol-prefetchedProjectsBlob'}).get('content'))['projects']will fix this issue. This is due to the overleaf webpage is changed, and the original code cannot find projects anymore. Maybe I will make a pull request later.Thank you for your response. The command ols list executed successfully, however, the command 'ols download -n "test"' did not. It returned the following error message:
$ ols download -n test 💥 Querying project 💥 Downloading project's PDF Error: Downloading project's PDF failed. Please try again.I would appreciate it if you could look into this issue further. Thanks in advance.
Change olclient.py line 104 to BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': 'ol-prefetchedProjectsBlob'}).get('content'))['projects'] will fix this issue.
Change olclient.py line 92 to
BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': 'ol-prefetchedProjectsBlob'}).get('content'))['projects']will fix this issue. This is due to the overleaf webpage is changed, and the original code cannot find projects anymore. Maybe I will make a pull request later.Thank you for your response. The command ols list executed successfully, however, the command 'ols download -n "test"' did not. It returned the following error message:
$ ols download -n test 💥 Querying project 💥 Downloading project's PDF Error: Downloading project's PDF failed. Please try again.I would appreciate it if you could look into this issue further. Thanks in advance.
Change olclient.py line 104 to BeautifulSoup(projects_page.content, 'html.parser').find('meta', {'name': 'ol-prefetchedProjectsBlob'}).get('content'))['projects'] will fix this issue.
Sorry for the delayed reply. It is good to know this issue has been fixed!