exmld

This application allows Kinesis and DynamoDB streams to be processed using Elixir or Erlang (by way of the KCL MultiLangDaemon). It's particularly useful when aggregate records are being used and items can be processed in approximate order (as opposed to strict order within each shard), but that isn't a requirement.

Using erlmld, a normal Erlang Kinesis processing application looks like this:

Using this Elixir library (which uses erlmld), a processing application looks like this:

This is done using the Flow framework to set up a MapReduce-style processing pipeline within a single BEAM node.

By virtue of using the KCL, processing applications can horizontally scale across a group of (homogenous) worker instances.

Unlike most applications using the KCL's MultiLangDaemon, an Erlang or Elixir processing application using this library can easily make full use of a worker's processing power (even if the stream contains a single shard) due to use of the Flow framework.

Examples

See:

1. example erlang processor
2. example elixir processor