awesome-python icon indicating copy to clipboard operation
awesome-python copied to clipboard

Add Quix Streams (stream processing w Kafka)

Open merlin-lacuna opened this issue 2 years ago • 0 comments

What is this Python project?

Quix Streams is a library for developing real-time streaming applications focused on time-series data and high-performance. It was developed at McLaren for processing huge sensor data streams from F1 racing cars, and is designed to be used for high-frequency telemetry services when you need to process high volumes of time-series data with up to nanosecond precision. It uses a message broker such as Apache Kafka (instead of a database) so you can process time-series data with high performance and resource savings.

Using Quix Streams, you can:

  • Produce time-series and event data to a Kafka Topic.
  • Consume time-series and event data from a Kafka Topic.
  • Process data by consuming it from one Kafka Topic, process it, and then producing the results back to another Kafka Topic.
  • Attach metadata to streams and group streams by metadata attributes.

What's the difference between this Python project and similar ones?

What's distinct about Quix Streams is its processing API—although it’s a client library, it’s incredibly powerful.

  • On the feature spectrum, it lies somewhere between Kafka client libraries such as Faust and kafka-python (which are easy to use but have a basic feature set) and full-fat stream processing engines like Flink and Spark (which are powerful but have a steep learning curve).
  • Includes many built in functions and data types such as ParameterData (for time-series data) and EventData, buffering and state management for rolling window calculations, serialization support for video and audio streams, and more.
  • The main difference is that processing is done in Python rather than Java, and it runs in a client environment rather than a server-side engine. --

Anyone who agrees with this pull request could submit an Approve review to it.

merlin-lacuna avatar Feb 20 '23 05:02 merlin-lacuna