Using Proxy

[shing-li]

trafficstars

There's a example code for proxy, but I didn't see the proxy parameter in class AsyncWebCrawler()

[unclecode]

Hi, This is code sample to try proxy:

async def use_proxy():
    print("\n--- Using a Proxy ---")
    print(
        "Note: Replace 'http://your-proxy-url:port' with a working proxy to run this example."
    )
    # Uncomment and modify the following lines to use a proxy
    async with AsyncWebCrawler(verbose=True, proxy="http://your-proxy-url:port") as crawler:
        result = await crawler.arun(
            url="https://www.nbcnews.com/business",
            bypass_cache=True
        )
        print(result.markdown[:500]) 

Feel free to let me know if you need any thing :)

[PratikHotchandani22]

Hi @unclecode, I am getting this error after running the above code:

AsyncWebCrawler.init() got an unexpected keyword argument 'proxy'

[anthonytison]

Hi @unclecode, I am getting this error after running the above code:

AsyncWebCrawler.init() got an unexpected keyword argument 'proxy'

I got the same issue :(

[PratikHotchandani22]

I can see an active PR regarding this issue:

https://github.com/unclecode/crawl4ai/pull/119

[SwallowData]

TypeError: AsyncWebCrawler.init() got an unexpected keyword argument 'proxy' me too

[aravindkarnam]

Here's the documentation on using proxy