crawl4ai
crawl4ai copied to clipboard
[Bug]: Webhook API within Docker missing CrawlerConfig/Browser config
crawl4ai version
latest
Expected Behavior
I successfully build a crawler request with BrowserConfig and CrawlerConfig with CSSExtraction, etc. Now I want to build a webhook strategy to not wait for the crawler response so I used this from the documentation https://docs.crawl4ai.com/core/docker-deployment/#job-queue-webhook-api
Current Behavior
The request body of /crawl/job does not accept BrowserConfig and CrawlerConfig as /crawl does. How can I define them?
Is this reproducible?
Yes
Inputs Causing the Bug
const crawler_config_payload: CrawlerRunConfig = {
type: 'CrawlerRunConfig',
params: {...}
}
const crawl_payload = {
urls: searchURLs,
browser_config: browser_config_payload,
crawler_config: crawler_config_payload,
}
const response: {
results: {
extracted_content: string
}[]
} = await $fetch(`${config.crawl4AiUrl}/crawl/job`, {
method: 'post',
body: crawl_payload,
})
Steps to Reproduce
Code snippets
OS
macOS
Python version
DOcker
Browser
No response
Browser version
No response
Error logs & Screenshots (if applicable)
No response
Any answer ? I want to introduce webhooks cause my request to the docker API is getting a timeout after, 60000. I tried to increase the timeout in my Axios client and the timeout of crawl4ai, but it does not work. I'm also open to other approaches to prevent the timeout.