crawl4ai icon indicating copy to clipboard operation
crawl4ai copied to clipboard

[Bug]: Docker API JSON serialization fails for ProxyConfig

Open SohamKukreti opened this issue 2 weeks ago • 1 comments

crawl4ai version

0.7.7

Expected Behavior

When using proxy_config with BrowserConfig, the configuration should be serializable to JSON for use with the Docker API server's crawler pool.

The BrowserConfig.to_dict() methods should properly serialize ProxyConfig objects by calling their respective to_dict() methods.

Current Behavior

When passing proxy_config to BrowserConfig, the Docker API server crashes with:

TypeError: Object of type ProxyConfig is not JSON serializable

This occurs in crawler_pool.py when trying to generate a configuration signature:

payload = json.dumps(cfg.to_dict(), sort_keys=True, separators=(",",":"))

The issue is that BrowserConfig.to_dict() directly assign ProxyConfig objects instead of converting them to dictionaries first.

Is this reproducible?

Yes

Inputs Causing the Bug

URL: https://example.com/
Request to the docker server:
{
        "urls": ["https://example.com"],
        "browser_config": {
            "headless": True,
            "proxy_config": {
                "server": "http://proxy.example.com:8080",
                "username": "user",
                "password": "pass"
            }
        }
    }

Steps to Reproduce

Send request to docker server with proxy_config in browser_config

Code snippets

# Reproduction code
import requests

response = requests.post(
    "http://localhost:11235/crawl",
    json={
        "urls": ["https://example.com"],
        "browser_config": {
            "headless": True,
            "proxy_config": {
                "server": "http://proxy.example.com:8080",
                "username": "user",
                "password": "pass"
            }
        }
    }
)

OS

Linux

Python version

3.10

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

[INIT].... → Crawl4AI 0.7.7 INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:11235 (Press CTRL+C to quit) 2025-11-21 20:15:19,049 - api - ERROR - Crawl error: Object of type ProxyConfig is not JSON serializable Traceback (most recent call last): File "/home/soham/Documents/crawl4ai/crawl4ai/deploy/docker/api.py", line 550, in handle_crawl_request crawler = await get_crawler(browser_config) File "/home/soham/Documents/crawl4ai/crawl4ai/deploy/docker/crawler_pool.py", line 36, in get_crawler sig = _sig(cfg) File "/home/soham/Documents/crawl4ai/crawl4ai/deploy/docker/crawler_pool.py", line 27, in _sig payload = json.dumps(cfg.to_dict(), sort_keys=True, separators=(",",":")) File "/usr/lib64/python3.10/json/init.py", line 238, in dumps **kw).encode(obj) File "/usr/lib64/python3.10/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib64/python3.10/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/lib64/python3.10/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.class.name} ' TypeError: Object of type ProxyConfig is not JSON serializable

SohamKukreti avatar Nov 21 '25 14:11 SohamKukreti