openai-cookbook
openai-cookbook copied to clipboard
crawl.py needs to parse "?" in filenames
trafficstars
Line 137 of crawl.py throws an exception when a question mark is in the URL. This character is not allowed in Windows file names. Replacing line 137 with something like the following resolves the issue.
# Save text from the url to a <url>.txt file
with open('text/'+local_domain+'/'+url[8:].replace("/", "_").replace("?","__") + ".txt", "w", encoding="UTF-8") as f: