`kafka-http` feature (webhooks)

[jfallows]

Kafka Webhooks mapping

• kafka-http binding handles mapping of kafka messages on a topic to a specific url endpoint
  • [ ] inject a consistent idempotency-key http header based on request hash if not present
  • [ ] inject http :scheme, :authority, and :path headers if not present
    • derived from url message key
  • [ ] requires tls to trust server certificates for all webhook subscribers, possibly just trustcacerts: true
  • [ ] requires tcp to support outbound connections to dynamic host name and port number
    • inject target address in stream metadata, much like dynamically discovered kafka broker hostnames
  • [ ] http client request security
    • request signing
    • injected authorization header via guard
  • [ ] retry logic for failed webhook requests
    • should pause delivery for that endpoint to maintain webhook event ordering
    • should not pause delivery for other endpoints, they should proceed in parallel
    • behaves as if multiple consumers over shared messages topic, filtered by consumer to give a virtual topic
    • each virtual topic is a stream of messages with the same key, from the same partition
    • consumer marks progress on the virtual topic, manage ack gaps to prevent redelivery of ack'd virtual topic messages
    • discover virtual topics for system topic consumer group member
      • adjusts dynamically during consumer group rebalance
      • consumer id implicitly derived from underlying consumer group id and message key
  • [ ] capture latest webhook response in correlated reply topic using same key as request message
    • used to verify behavior during setup
    • includes attempt number as kafka message header
    • give up when maximum retry attempts exceeded
  • [ ] expire messages using time-based retention policy on requests topic to prevent retries, (say) after 24h

Note: monitor via http metrics at http client binding, export via prometheus, etc - may require high cardinality dynamic metrics.

Note: resolution of micro service events to filtered webhook subscribers is applied before publishing subscriber url targeted messages to the kafka topic.

Note: receiving inbound webhooks is handled by http-kafka binding, may need minor additional work to support full inbound url as produced message key, or to restrict set of headers included with request message payload.

Note: broadcast scenarios where one message is sent to multiple webhook endpoints requires a late join of request with different urls, which is anticipated to have occurred before zilla observes the webhook request in the kafka requests topic.

[arnitolog]

This is a brilliant feature. It will be a real game-changer. It will allow to turn legacy products with REST APIs only into the modern event-driven architecture. Looking forward to it.