crawl4ai

crawl4ai copied to clipboard

Refactor extraction and completion functions to support asynchronous …

1

…execution ## Summary Refactor extraction and completion functions to support asynchronous execution Fixes #1055 ## List of files changed and why - async_webcrawler.py - content_filter_stratery.py - extraction_strategy.py - utils.py ##...

Ahmed-Tawfik94

feat(crawler): add content change detection and timestamp management

1

## Summary This PR introduces a new feature for cache management - content change detection. Users can now specify whether they want to check if web content has changed before...

yesidc

Update utils.py fix get_title

1

Models & Pricing | DeepSeek API Docs The above html code does not get the title ## Summary by CodeRabbit - **Bug Fixes** - Improved extraction of page titles from...

lijc210

fix(extraction): llm extraction strategy not overriding response

1

## Summary In extract() of LLMExtractionStrategy, `response` is not overridden. Thus if some error is met in later part of this `try`, in `except` we'll get error triggering response.choices[0].message.content: 'str'...

scris

[Bug]: crawl4ai爬取pdf网页出现问题│ Error: 'PDFContentScrapingStrategy' object has no attribute 'run'

2

### crawl4ai version 0.6.2 ### Expected Behavior [FETCH]... ↓ http://ayacd.yrdcpcn.org.cn/upload/2023/0807/b316d... | Status: False | Time: 0.28s [SCRAPE].. ◆ http://ayacd.yrdcpcn.org.cn/upload/2023/0807/b316d... | Time: 0.0s [ERROR]... × http://ayacd.yrdcpcn.org.cn/upload/2023/0807/b316d... | Error: ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ×...

zhoufei0622

[Bug]: Get the same results for different URLs and Failed on navigating ACS-GOTO with multiple urls crawling

1

### crawl4ai version 0.6.3 ### Expected Behavior Getting the data of each link correctly and not getting the Failed on navigating ACS-GOTO with multiple URLs crawl. I tried using a...

Auth0rM0rgan

[Bug]: IndexError: list index out of range when accessing context.pages[0] with use_managed_browser enabled

3

### crawl4ai version 0.6.3 ### Expected Behavior NA ### Current Behavior When running with use_managed_browser=True, the crawler crashes with an IndexError: list index out of range at line 967 in...

aravindkarnam

feat: Add prompt-driven recursive crawler script

2

This commit introduces a new example script, `prompt_driven_crawler.py`, located in `docs/examples/`. The script enables you to perform a recursive crawl starting from a given URL. It uses an LLM (OpenAI...

tmocky1134

enforce auth when jwt is enabled

1

## Summary If no credentials are provided, the token gets never verified, even if jwt is enabled. This PR throws a 401 exceptions then. ## List of files changed and...

chrizzly2309

### crawl4ai version 0.7.7 ### Expected Behavior When I use the latest version of Docker deployment and set 'max_dession_permit' to 1, an error message will appear, as follows: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File...

liuHongyang0207