ultimate-sitemap-parser icon indicating copy to clipboard operation
ultimate-sitemap-parser copied to clipboard

add optional argument to requests web client, to ignore SSL checking

Open japherwocky opened this issue 3 years ago • 0 comments

Hello -

This adds a very simple verify argument to the requests based web client. You can use it pretty easily for issues like #33 by doing something like:

        from usp.tree import sitemap_tree_for_homepage
        from usp.web_client.requests_client import RequestsWebClient

        client = RequestsWebClient(verify=False)  # this PR adds the verify kwarg, which defaults to True
        client.__USER_AGENT = 'Friendly Spider /{}'

        tree = sitemap_tree_for_homepage('https://nytimes.com', client)

japherwocky avatar Dec 23 '22 00:12 japherwocky