Seamless Microservices RFC

[javiertoledo]

Request For Comments/Design Proposal for a Seamless Microservices feature in Booster

TL;DR: We want to design a seamless microservices interaction between Booster services and any external source. Our goal is that we achieve a developer experience that feels like events teleportation!

This proposal explores a design strategy to build Microservice architectures with Booster. Booster applications are, at the end of the day, just node applications running on FaaS services, so it’s currently possible to use standard event transport mechanisms like Kafka, RabbitMQ, EventBridge, and others. So, this document does not discuss if it’s possible, but how to build a seamless “Booster experience” on the line of how we handle event-sourcing or API generation within Booster nowadays.

Booster is an event-driven framework, and we’re fully convinced of the benefits of event-driven microservices architectures, as we’ve experienced first-hand the problems of API-driven microservices architectures at scale, so we’re also focusing on an event-driven solution. We're designing here a mechanism in which one or more Booster applications participate in a wider, likely heterogeneous multi-service system in which many services interact with one or more central event transport mechanisms in a loosely coupled way (Event-Driven Microservices Architecture).

Taking into account this scenario, we define the following design drivers for what we call "a seamless Booster experience for Microservices":

• Event-driven: Services communicate via events.
• Embeddable: It can use existing instances of commonly used event transport mechanisms (i.e. Apache Kafka, EventBridge, RabbitMQ, etc.) that are not part of the current application stack.
• Adaptable: It can be easily adapted to any other technologies by implementing integration modules (similar design to the current providers implementation)
• Compatible: Can be integrated into heterogeneous architectures in which each service can be implemented in any other technology.
• Maintainable: Booster will be able to properly migrate over schema changes.
• High-level of abstraction: As other features in Booster, integration should be implicit and highly semantic, with minimal or no code at all.

With these ideas in mind here are some architecture and developer-experience proposals:

Streamer connector interface

We define a “streamer” as any mechanism capable of transporting and or storing events. Generalizing, something that can either:

• Accept events with an operation that we will call publish.
• Provide access to events previously submitted.

It’s common to organize events into topics or streams, so we’ll provide an optional "topic" parameter when publishing or reading events as a level of event organization, but all other details like event ordering, consistency of writes/reads or partitioning will be responsibility of the specific implementation used.

Publishing events is modeled as a synchronous operation like an API or RPC call, but integration options differ when we take into account how events are read. We will initially focus on two integration scenarios:

• Pull (Polling): The consumer process polls the streamer (i.e. A self-managed Apache Kafka cluster).
• Push (via FaaS integration): The integration is made at cloud provider configuration level, setting a Booster function to be triggered by the cloud provider when new data is available (I.e. Amazon EventBridge, Azure EventHub, or even some provider-managed Kafka solutions)

In order to connect a Booster service to a streamer, it will be necessary to implement a streamer connector, which is a module that matches the StreamerConnector interface initially defined as follows:

import { EventInterface } from '@boostercloud/framework-types'

/** Wrapper for an event received from an external event streamer */
interface ExternalEvent {
  streamerId: string
  streamerTopic?: string
  /** Events that match the Booster event format whose types are available will be transformed into Booster events
   * otherwise, they'll be added in their original format.
   */
  eventPayload: EventInterface | unknown
}

/** Interface for a streamer connector module. It defines the methods to connect to an event streaming service.
 * There are different strategies to receive events from streaming services:
 * - Polling: The connector will periodically check for new events in the streaming service
 * - FaaS: A lambda will be triggered anytime there are new events in the streaming service
 */
export type StreamerConnector = StreamerConnectorPublishInterface &
  (StreamerConnectorPollingInterface | StreamerConnectorFaaSInterface)

/** The publish method is defined the same way for all streaming connectors regardless of the receiving strategy */
interface StreamerConnectorPublishInterface {
  /** Publishes one or more events in the specified topic */
  publish(events: EventInterface[], topic?: string): Promise<void>
}

/** Polling strategy: Booster will call this method periodically to check for new events in the streaming service */
interface StreamerConnectorPollingInterface {
  /** It must connect to the streaming service, fetch the new events and parse them using the `ExternalEvent` wrapper */
  poll(topic?: string, lastReceivedEventID?: string): Promise<ExternalEvent[]>
}

/** FaaS strategy: Booster will trigger this method when the notification lambda is triggered */
interface StreamerConnectorFaaSInterface {
  /** Events are injected by the streaming service in their propietary format, so the connector must parse them using the `ExternalEvent` wrapper  */
  transformEvents(events: unknown[], topic?: string): Promise<ExternalEvent[]>
}

Registering streamer connectors in the configuration

The user will be able to define as many streamers as needed in a new Booster.config.streamers entry that will accept an object where the keys are the ids of the streamers or streamerId (chosen by the user), and the values are streamer connector instances. This will allow the system to easily reference any streamer by this ID.

import { KafkaStreamer } from '@boostercloud/kafka-streamer'
import { EventBridgeStreamer } from '@boostercloud/eventbridge-streamer'

...
config.streamers = {
  'main-kafka': KafkaStreamer.with(/* Kafka-specific configuration parameters */)
  'event-bridge': EventBridgeStreamer.with(/* EventBridge-specific configuration */)
}

Setting an event to be streamed

We will introduce a new streamTo parameter in the @Event decorator that will accept one or more streamer ‘URIs’ in the format <streamerId>/<topic-name> (Topics can be optional when the streamer implementation has a default topic or does not use topics at all). This parameter will accept one or an array of streamers URIs. When defined, the events will be automatically published to the chosen streamers and topics after they’re properly stored in the Booster event store:

// Event automatically streamed to the `main-kafka` streamer in the `posts` topic
@Event({ streamTo: 'main-kafka/posts' }) 
export class PostCreated {
  public constructor(
    readonly postId: UUID,
    readonly title: string,
    readonly content: string,
    readonly author: string
  ) {}
}

// Event automatically streamed to both the `main-kafka` streamer in the `posts` topic and the `event-bridge` streamer in the topic `integrations`
@Event({ streamTo: ['main-kafka/posts', 'event-bridge/integrations' }) 
export class PostCreated {
  public constructor(
    readonly postId: UUID,
    readonly title: string,
    readonly content: string,
    readonly author: string
  ) {}
}

Booster will handle the publication, using the corresponding event streamer implementation upon event registration without further interaction of the user.

Receiving events

Likewise, thanks to the abstraction provided by the streamer implementation, receiving external events will be also integrated seamlessly with the Booster development flow. You can receive external events from the streamers by just adding a streamFrom parameter to an @EventHandler decorator:

import { PostCreated } from '@application-name/main-kafka-types'

// When importing Booster generated events (from a different Booster service), 
// and we have the event schema available (i.e. in a shared node module),
// The event is automatically loaded with the right Booster event type.
@EventHandler({ eventType: PostCreated, streamedFrom: 'main-kafka/posts' })
export class PostUpdates {
  public static async handle(event: PostCreated, register: Register): Promise<void> {
		// Make sure to do validation to make sure that the event is trustable
    Booster.logger.debug('PostCreated received for post ' + event.postId
  }
}

// When the event can't be parsed automatically
// it's unwrapped as an unknown object.
@EventHandler({ streamedFrom: 'main-kafka/posts' })
export class PostUpdates {
  public static async handle(event: unknown, register: Register): Promise<void> {
    // Make sure to do validation to make sure that the event is trustable and
    // can be properly parsed
    Booster.logger.debug('A raw event received: ' + event.aKnownField
  }
}

Additional details

This document describes the happy path for this feature, which is enough to trigger the discussion, but they could change as we work on it. Anyways, we want to highlight some points in advance that have been raised on initial conversations on the topic here.

Optimizing streamer interaction thanks to Booster code inspection capabilities

Notice that we’re not explicitly subscribing to specific stream services or topics. This doesn’t mean that we’re planning to be actively listening or polling every topic in a Kafka instance and filtering the events. Booster integrates an extension of the typescript compiler that is capable of extracting metadata from code, and we’re planning to use this feature to build a list of services and topics that are actually used in code to make sure that we’re listening to them and nothing more. This has a side advantage: If a topic is removed from an event handler, the link will be removed automatically too.

Event schemas and migrations

Booster has an event migrations mechanism that allows events to be upgraded on the fly when an old version of an event is found in the database. We envision that in microservices architectures where all services are built with Booster, the teams that interact have access to one or more shared TypeScript libraries where they maintain both the event schemas and the migration classes. In this way, when a new version is released, all dependent services will fail at compile time, which is something we can use in CI/CD processes to avoid deploying a new version of a service that will make dependent services fail. We’ve found this approach to be extremely useful as an organization scales in number of services and teams.

For other scenarios, for example, when a Booster application is receiving events via Kafka from, let’s say, a Kotlin application that is encoding the data in AVRO format and putting the schemas in Schema Registry, we fall back to the streamer connector implementation. AVRO decoding and interaction with Schema registry can be implemented as part of the streamer connector implementation, and if the developer is generating TypeScript classes from the schema registry, the streamer connector could even generate a valid Booster event and trigger the event handler as if the event came from a Booster service. At this point it’d be naive to affirm that the proposed developer experience is generalizable, so we're keeping the option to receive events of type unknown, but we believe that the proposed solution is flexible enough to allow streamer connector developers to come up with really interesting solutions that level up the final experience for their users.

[adrian-lorenzo]

This is an awesome proposal @javiertoledo! It matches perfectly the needs while keeping the framework principles.

I have a minor concern for the user experience: I would find a way to avoid using literal strings for the streamTo attribute, to avoid problems with configuration (e.i. writing strem/kafka instead of stream/kafka). It is true that good programming practices should be enough to avoid problems with this, but I think it would be awesome to make it mandatory to link a proper Stream instance and not make the user choose how to deal with good practices.

[javiertoledo]

Maybe we can also introduce a helper function to type check the streamer name:

@Event({ streamTo: streamer('main-kafka', 'posts') })
...

What do you think @boostercloud/booster-core?