crawl4ai icon indicating copy to clipboard operation
crawl4ai copied to clipboard

Fix crawling error in AsyncWebCrawler

Open theguy000 opened this issue 1 year ago • 0 comments

Related to #105

Fix the 'NoneType' object has no attribute 'get' error in AsyncWebCrawler.

  • crawl4ai/async_webcrawler.py

    • Add a check in the arun method to ensure html is not None before further processing.
    • Raise a descriptive error if html is None.
  • crawl4ai/async_crawler_strategy.py

    • Add a check in the crawl method of the AsyncPlaywrightCrawlerStrategy class to handle cases where html is None.
    • Raise a descriptive error if html is None.
  • tests/async/test_basic_crawling.py

    • Add a test case to verify handling of None values for the html variable in the test_invalid_url function.
  • tests/async/test_error_handling.py

    • Add a test case to verify handling of None values for the html variable in the test_network_error function.

For more details, open the Copilot Workspace session.

theguy000 avatar Oct 03 '24 05:10 theguy000