Website Q & A bot crawler TooManyRedirects

[Arnasltlt]

I am not too qualified to assess this, but in the Q&A crawler example https://github.com/openai/openai-cookbook/tree/main/solutions/web_crawl_Q%26A I was encountering this error: requests.exceptions.TooManyRedirects: Exceeded 30 redirects

ChatGPT solved it for me by including this in the crawler function:

            except requests.exceptions.TooManyRedirects:
                print("Too many redirects for " + url + ", moving on to next link.")
                continue

Maybe good to include it