TransactionalBox

TransactionalBox copied to clipboard

:star: - The star motivates me a lot!

Transactional box is an implementation of the outbox and inbox pattern in .NET.
Ensures eventual consistency when modules need to communicate with each other over the network.

Examples of problems that occur during network communication:

• lost messages
• the same messages were processed again
• unavailable services

✨ Features

Actions

• [x] Add a message to send to the outbox
• [ ] Add a message to publish to the outbox
• [x] Get messages from outbox and add them to transport
• [x] Get a message from transport and add them to the inbox
• [x] Get a message from inbox and process it
• [ ] Get messages from inbox and process them

Storage

• [x] Entity Framework (Relational)
  • [x] Migrations
  • [X] Distributed Lock (Based on atomic write operation, Standalone Package)
• [x] InMemory
• [ ] MongoDB

Transport

• [x] Apache Kafka
• [x] InMemory
• [ ] RabbitMQ
• [ ] Iggy
• [ ] HTTP
• [ ] gRPC

Scalability & Fault Tolerance

• [x] Support for multiple outbox worker instances
  • [x] Multiple instances of the same service
  • [x] Multiple processes in the same service
• [x] Support for multiple inbox worker instances
  • [x] Multiple instances of the same service
  • [x] Multiple processes in the same service
• [x] Support for multiple inbox instances
  • [x] Multiple instances of the same service
  • [x] Multiple processes in the same service
• [ ] Standalone outbox worker
• [ ] Standalone inbox worker
• [x] Error handling in background services
• [ ] Task throttling in background services
• [ ] Dead messages

Observability

• [ ] Support for OpenTelemetry
• [ ] Outbox size
• [ ] Inbox size
• [ ] Message failure rate
• [ ] Message delivery latency
• [ ] Number of duplicated messages
• [ ] Message duplication rate

Maintenecne

• [x] Remove processed messages from the outbox
• [x] Remove processed messages from the inbox
• [x] Remove expired idempotency keys
• [ ] Archive processed messages from the outbox
• [ ] Archive processed messages from the inbox
• [x] Correlation ID

Other

• [x] Modular package architecture
• [x] Support for TimeProvider
• [x] Unordered messages
• [X] Internal high-performance logging (Source Generators)
• [x] Execution context in Inbox
• [x] Grouping of messages
  • [x] Group by topic outbox messages to a single transport message from batch (better compression)
  • [x] Adjusting optimal transport message size
• [X] Messages serialization and deserialization
  • [X] System.Text.Json (default)
  • [x] Custom (you can add your own implementation)
• [x] Messages compression and decompression
  • [X] No compression (default)
  • [X] Brotli
  • [x] GZip
  • [x] Custom (you can add your own implementation)
• [ ] Message streaming
• [ ] Package configuration using appsetings.json
• [ ] Own transport message serialier and deseralizer (with StringBuilder)
• [x] Idempotent messages
• [x] Keyed in memory lock (based on SemaphoreSlim and ConcurrentDictionary)

:clapper: Run Sample

[!NOTE] Docker is required.

Clone this repo and open TransactionalBox.sln via Visual Studio 2022. Set the TransactionalBox.Sample.WebApi as startup and then run. You should see the following view.

Have fun :smiley:!

:european_castle: Architecture

The transactional box consists of four basic components. The following diagrams show the basic flow (omits details).

Outbox

Outbox is responsible for adding messages to the storage.

Outbox Worker

Outbox worker is responsible for getting the messages from storage and adding them to the transport.

Inbox Worker

Inbox worker is responsible for getting messages from transport and adding them to the storage.

Inbox

Inbox is responsible for processing messages from the storage.

:medal_sports: Competition '100commitow'

The project is part of the competition 100 commitow.

Topics

• Distributed lock (prevent race condition)
• Concurrency control
• Scaling and parallel processing
• Synchronization primitives
• Idempotency
• Retry Pattern