crawl4ai
crawl4ai copied to clipboard
Make LLM backoff configurable end-to-end
Summary
This PR:
- Extends LLMConfig with backoff delay/attempt/factor fields and thread them through LLMExtractionStrategy, LLMContentFilter, table extraction, and Docker API handlers
- Exposes the backoff parameter knobs on perform_completion_with_backoff/aperform_completion_with_backoff and document them in the md_v2 guides
Fixes: #1269
List of files changed and why
crawl4ai/async_configs.py– Added configurable backoff fields to LLMConfig.
crawl4ai/content_filter_strategy.py – Thread new backoff values into LLMContentFilter.
crawl4ai/extraction_strategy.py – Pass configurable delays/retries to LLMExtractionStrategy.
crawl4ai/table_extraction.py – Ensure table extraction LLM calls respect the new backoff config.
crawl4ai/utils.py – Updated perform_completion_with_backoff (sync/async) signature and logic.
deploy/docker/api.py – Allow Docker LLM endpoints to accept custom backoff settings.
docs/md_v2/api/parameters.md, docs/md_v2/core/browser-crawler-config.md, docs/md_v2/complete-sdk-reference.md – Documented the new LLMConfig options and examples.
How Has This Been Tested?
Ran the LLM extraction example in a custom script with and without custom backoff parameters to ensure behavior is unchanged by default and configurable when supplied.
Ran the LLM content filter example (custom vs default backoff) to confirm the new values propagate through LLMContentFilter.
Started the Docker API locally and exercised the new Example 6 (LLM extraction via /crawl/job) with and without custom backoff payloads to verify the REST surface.
Spot-checked other examples/tests that rely on LLMConfig to ensure defaults still produce successful runs.
Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] I have added/updated unit tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes