opencode icon indicating copy to clipboard operation
opencode copied to clipboard

feat: experimental durable streaming architecture

Open joelhooks opened this issue 3 weeks ago • 0 comments

    ╔═══════════════════════════════════════════════════════════════╗
    ║                                                               ║
    ║   ██████╗ ██╗   ██╗██████╗  █████╗ ██████╗ ██╗     ███████╗   ║
    ║   ██╔══██╗██║   ██║██╔══██╗██╔══██╗██╔══██╗██║     ██╔════╝   ║
    ║   ██║  ██║██║   ██║██████╔╝███████║██████╔╝██║     █████╗     ║
    ║   ██║  ██║██║   ██║██╔══██╗██╔══██║██╔══██╗██║     ██╔══╝     ║
    ║   ██████╔╝╚██████╔╝██║  ██║██║  ██║██████╔╝███████╗███████╗   ║
    ║   ╚═════╝  ╚═════╝ ╚═╝  ╚═╝╚═╝  ╚═╝╚═════╝ ╚══════╝╚══════╝   ║
    ║                                                               ║
    ║   ███████╗████████╗██████╗ ███████╗ █████╗ ███╗   ███╗███████╗║
    ║   ██╔════╝╚══██╔══╝██╔══██╗██╔════╝██╔══██╗████╗ ████║██╔════╝║
    ║   ███████╗   ██║   ██████╔╝█████╗  ███████║██╔████╔██║███████╗║
    ║   ╚════██║   ██║   ██╔══██╗██╔══╝  ██╔══██║██║╚██╔╝██║╚════██║║
    ║   ███████║   ██║   ██║  ██║███████╗██║  ██║██║ ╚═╝ ██║███████║║
    ║   ╚══════╝   ╚═╝   ╚═╝  ╚═╝╚══════╝╚═╝  ╚═╝╚═╝     ╚═╝╚══════╝║
    ║                                                               ║
    ╚═══════════════════════════════════════════════════════════════╝

Why

SSE connections drop. Networks flake. Clients reconnect. When they do, they miss events.

Durable streams solve this: persist events with monotonic offsets, let clients catch up from where they left off. No lost messages, no polling, no complexity on the client side.

How

┌─────────────────────────────────────────────────────────────┐
│                        DATA FLOW                            │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│   Bus.publish(event)                                        │
│         │                                                   │
│         ▼                                                   │
│   EventStore.append() ──► SQLite + ULID offset              │
│         │                                                   │
│         ▼                                                   │
│   GET /stream/events?offset=X                               │
│         │                                                   │
│         ├──► Catch-up: JSON array of missed events          │
│         └──► Live: SSE stream from offset onwards           │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Config-gated. Zero behavior change when disabled:

// opencode.jsonc
{
  "experimental": {
    "durableStreams": true
  }
}

What's in the box

Component File Purpose
Config flag config.ts experimental.durableStreams (default: false)
EventStore event-store/ SQLite persistence, ULID offsets, catch-up queries
ServerRegistry server/registry.ts JSON file-based multi-server discovery
Stream endpoint server/stream.ts GET /stream/events (catch-up + live SSE)
Discovery server/discovery.ts GET/POST/DELETE /servers
Persistence hook bus/index.ts Auto-persist on Bus.publish()
Self-registration cli/cmd/serve.ts 15s heartbeat, graceful shutdown

Offset semantics

  • Offsets are opaque ULID strings (lexicographically sortable)
  • offset="-1" = start from beginning
  • Response includes Stream-Next-Offset header for resumption
  • Client stores offset, reconnects with it, gets exactly what they missed

Tests

407 pass | 0 fail

Full coverage on EventStore and ServerRegistry with edge cases for catch-up semantics, payload robustness, and ULID monotonicity.

joelhooks avatar Dec 29 '25 18:12 joelhooks