[Bug]: httpcore.PoolTimeout during vectorization: HTTP connection pool exhaustion with large documents

[hxu296]

What did you do when it broke?

Uploaded a large PDF document (~140 pages) to Open Notebook and enabled vectorization for embedding. The document was split into 2,573 chunks for processing.

How did it break?

The vectorization process failed after processing the first 808 chunks. The remaining 1,765+ chunks encountered httpcore.PoolTimeout errors because all 2,573 chunks were being processed concurrently, exceeding httpx's connection pool limit of 100.

The asyncio.gather(*tasks) call in Source.vectorize() attempted to process all chunks simultaneously, creating thousands of concurrent HTTP requests to the Ollama embedding API. After the first 100 connections were established, the remaining requests queued and timed out after 60 seconds.

Logs or Screenshots

<AsyncConnectionPool [Requests: 100 active, 1648 queued | Connections: 100 active, 0 idle]>

httpcore.PoolTimeout File "/httpcore/_async/connection_pool.py", line 256, in handle_async_request raise exc from None

2025-10-29 00:03:24.250 | ERROR | open_notebook.domain.notebook:process_chunk:310 - Error processing chunk 808: Failed to get embeddings: ... (repeated for chunks 809-2573)

Open Notebook Version

Other (please specify in additional context)

Environment

OS: macOS Package Manager: uv Embedding Model: custom Ollama model

Additional Context

open-notebook version: Commit: bc35a9511775 (main branch)

[hxu296]

I worked around this issue by implementing batch processing in open_notebook/domain/notebook.py. Instead of processing all chunks concurrently, I processed in batches of 50 chunks and serialized between batch. This will fix embedding for large documents.

[lfnovo]

This should be fixed now. Pushed the vectorization to be a background task, running chunks individually and implementing a proper retry mechanism for failures. Please give it a shot.