🐛 [Bug]: 409 Conflict Spam When Processing a Crawl

[phill-bramble]

Archon Version

main pull as of 20/08/2025

Bug Severity

🟡 Medium - Affects functionality

Bug Description

My server log is full of 409 Conflict errors when processing crawls.

Steps to Reproduce

Crawl any site

Expected Behavior

Successful posts to Supabase

Actual Behavior

2025-08-20 15:48:51 | httpx | INFO | HTTP Request: POST http://host.docker.internal:8000/rest/v1/archon_crawled_pages⁠ "HTTP/1.1 409 Conflict"

2025-08-20 15:48:51 | search | ERROR | Failed individual insert for https://nextjs.org/docs/app/api-reference/functions/unstable_noStore:⁠ {'message': 'duplicate key value violates unique constraint "archon_crawled_pages_url_chunk_number_key"', 'code': '23505', 'hint': None, 'details': 'Key (url, chunk_number)=(https://nextjs.org/docs/app/api-reference/functions/unstable_noStore, 7) already exists.'}

2025-08-20 15:48:51 | search | INFO | Individual inserts: 12/25 successful

Error Details (if any)

Affected Component

🔍 Knowledge Base / RAG

Browser & OS

N/A

Additional Context

No response

Service Status (check all that are working)

• [x] 🖥️ Frontend UI (http://localhost:3737)
• [x] ⚙️ Main Server (http://localhost:8181)
• [x] 🔗 MCP Service (localhost:8051)
• [x] 🤖 Agents Service (http://localhost:8052)
• [x] 💾 Supabase Database (connected)

[Wirasm]

We still have this issue by design, we should catch duplicates earlier in the crawling process. keeping in backlog for now as it does not break any critical flows

thank you for reporting this @phill-bramble