Streamops

StreamOps platform is a discovery, analytics and operations toolset enabling developers and businesses to control multiple Kafka, RedPanda and Kinesis clusters, export custom metrics, gain insights into their topologies and build pipelines.

StreamOps is an open source free product.

See changelog.

Distribution

You can clone this repo and build from source or use our docker image.

How to run

Streamops requires a database for persisting settings and connection details.

Launch the StreamOps application using the following command:

docker run streamops:streamops -p 39854:39854 -e STREAMOPS_JDBC_URL=<connection string>

Where connection string is a JDBC format url pointing to your database, for example, jdbc:mysql://db.myhost:3306/database?user=abc&password=xyz. Supported JDBC drivers are MySQL and Postgres. If you require more, open a ticket.

The username and password are optional, and can be specified using STREAMOPS_JDBC_USERNAME and STREAMOPS_JDBC_PASSWORD.

If the STREAMOPS_JDBC_URL environment variable is missing, then streamops will default to an ephemeral database. This is useful for evaluating StreamOps locally.

Using secrets

When you want to pass a password or other secret to a data source, you can use strings of the form:

• env://key -> to load an environment variable
• aws://key -> to resolve the key from AWS Secrets Manager.

Datasources

Streamops supports multiple datasources. You can add as many Kafka or Redpanda instances as you wish - for example for different environments like prod, staging, QA, etc.

Each datasource will then show the topology of the cluster.

Healthcheck

A unique feature of StreamOps is the healthchecks that you can run cluster wide. These will give you details of specific issues, such as lagging consumers, skewed partitions, non-preferred leaders and so on.

For example, this healthcheck will indicate any consumer groups with an unequal distribution of consumers.