crawl4ai
crawl4ai copied to clipboard
Fix crawling error in AsyncWebCrawler
Related to #105
Fix the 'NoneType' object has no attribute 'get' error in AsyncWebCrawler.
-
crawl4ai/async_webcrawler.py
- Add a check in the
arunmethod to ensurehtmlis notNonebefore further processing. - Raise a descriptive error if
htmlisNone.
- Add a check in the
-
crawl4ai/async_crawler_strategy.py
- Add a check in the
crawlmethod of theAsyncPlaywrightCrawlerStrategyclass to handle cases wherehtmlisNone. - Raise a descriptive error if
htmlisNone.
- Add a check in the
-
tests/async/test_basic_crawling.py
- Add a test case to verify handling of
Nonevalues for thehtmlvariable in thetest_invalid_urlfunction.
- Add a test case to verify handling of
-
tests/async/test_error_handling.py
- Add a test case to verify handling of
Nonevalues for thehtmlvariable in thetest_network_errorfunction.
- Add a test case to verify handling of
For more details, open the Copilot Workspace session.