crawl4ai
crawl4ai copied to clipboard
[Bug]: Docker API JSON serialization fails for ProxyConfig
crawl4ai version
0.7.7
Expected Behavior
When using proxy_config with BrowserConfig, the configuration should be serializable to JSON for use with the Docker API server's crawler pool.
The BrowserConfig.to_dict() methods should properly serialize ProxyConfig objects by calling their respective to_dict() methods.
Current Behavior
When passing proxy_config to BrowserConfig, the Docker API server crashes with:
TypeError: Object of type ProxyConfig is not JSON serializable
This occurs in crawler_pool.py when trying to generate a configuration signature:
payload = json.dumps(cfg.to_dict(), sort_keys=True, separators=(",",":"))
The issue is that BrowserConfig.to_dict() directly assign ProxyConfig objects instead of converting them to dictionaries first.
Is this reproducible?
Yes
Inputs Causing the Bug
URL: https://example.com/
Request to the docker server:
{
"urls": ["https://example.com"],
"browser_config": {
"headless": True,
"proxy_config": {
"server": "http://proxy.example.com:8080",
"username": "user",
"password": "pass"
}
}
}
Steps to Reproduce
Send request to docker server with proxy_config in browser_config
Code snippets
# Reproduction code
import requests
response = requests.post(
"http://localhost:11235/crawl",
json={
"urls": ["https://example.com"],
"browser_config": {
"headless": True,
"proxy_config": {
"server": "http://proxy.example.com:8080",
"username": "user",
"password": "pass"
}
}
}
)
OS
Linux
Python version
3.10
Browser
No response
Browser version
No response
Error logs & Screenshots (if applicable)
[INIT].... → Crawl4AI 0.7.7 INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:11235 (Press CTRL+C to quit) 2025-11-21 20:15:19,049 - api - ERROR - Crawl error: Object of type ProxyConfig is not JSON serializable Traceback (most recent call last): File "/home/soham/Documents/crawl4ai/crawl4ai/deploy/docker/api.py", line 550, in handle_crawl_request crawler = await get_crawler(browser_config) File "/home/soham/Documents/crawl4ai/crawl4ai/deploy/docker/crawler_pool.py", line 36, in get_crawler sig = _sig(cfg) File "/home/soham/Documents/crawl4ai/crawl4ai/deploy/docker/crawler_pool.py", line 27, in _sig payload = json.dumps(cfg.to_dict(), sort_keys=True, separators=(",",":")) File "/usr/lib64/python3.10/json/init.py", line 238, in dumps **kw).encode(obj) File "/usr/lib64/python3.10/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib64/python3.10/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/lib64/python3.10/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.class.name} ' TypeError: Object of type ProxyConfig is not JSON serializable