go-workflows icon indicating copy to clipboard operation
go-workflows copied to clipboard

Implement LISTEN/NOTIFY for Postgres backend reactive task polling

Open Copilot opened this issue 4 months ago • 0 comments

Workers currently poll Postgres at fixed intervals to check for new tasks. This adds latency and generates unnecessary database queries.

Changes

Database layer

  • Migration adds triggers on pending_events and activities tables that fire pg_notify() on insert
  • Notifications sent to workflow_tasks and activity_tasks channels with queue name as payload
  • Leverages existing unique index on (instance_id, execution_id) for efficient queue lookup

Notification listener (notify.go)

  • Manages dedicated pq.Listener connections for workflow and activity channels
  • WaitGroup-based synchronization prevents channel close races
  • Configurable reconnect intervals (10s-60s) and keepalive pings (90s)

Backend integration

  • GetWorkflowTask and GetActivityTask check for immediate work, then block on notification channel
  • Falls back to polling when notifications disabled
  • Dual driver approach: pgx for SQL operations, pq for LISTEN/NOTIFY

Configuration

  • WithNotifications(bool) option, disabled by default
  • Backward compatible—no changes required to existing code

Usage

b := postgres.NewPostgresBackend(
    "localhost", 5432, "user", "password", "db",
    postgres.WithNotifications(true),
)

Performance

Testing shows ~9x latency reduction:

  • Polling: 500-800ms task pickup latency
  • LISTEN/NOTIFY: 50-150ms task pickup latency

Sample application in samples/postgres-notify demonstrates the feature.

Original prompt

This section details on the original issue you should resolve

<issue_title>Use LISTEN/NOTIFY for Postgres queues</issue_title> <issue_description>Investigate if we can use LISTEN/NOTIFY for queues in the postgres backend.</issue_description>

Comments on the Issue (you are @copilot in this section)

  • Fixes cschleiden/go-workflows#455

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot avatar Nov 02 '25 21:11 Copilot