openai-cookbook icon indicating copy to clipboard operation
openai-cookbook copied to clipboard

crawl.py needs to parse "?" in filenames

Open bobthechemist opened this issue 2 years ago • 0 comments
trafficstars

Line 137 of crawl.py throws an exception when a question mark is in the URL. This character is not allowed in Windows file names. Replacing line 137 with something like the following resolves the issue.

# Save text from the url to a <url>.txt file
        with open('text/'+local_domain+'/'+url[8:].replace("/", "_").replace("?","__") + ".txt", "w", encoding="UTF-8") as f:

bobthechemist avatar Mar 18 '23 13:03 bobthechemist