KafkaDataGen icon indicating copy to clipboard operation
KafkaDataGen copied to clipboard

verified publisher icon appuv
→

Metadata

This repository provides Python scripts to generate simulated data and produce it into Kafka topics, facilitating testing and development of Kafka-based applications and pipelines.

Equipment Data Generation

Data Generation for the equipment stream analytics

CodeQL License GitHub top language

Prerequisite

  1. Python 3.8.5
  2. Docker
  3. Confluent Kafka 6.2.0

Python Packages Required :

confluent_kafka,kafka,kafka-python,requests,avro

Setup

Configure the config.properties

For creating mock avro records (Equipment Mock Streams)

python equipmentProduce.py

For creating meta json records (Meta Update Streams)

python equipmentConsume.py

Schema Evolution

For schema evolution change the config.properties to match the upgraded schema

equipment-value-schema=equipment-value-upgrade.avsc
mock_data=equipment-upgrade-mock

Further Reading

Medium

Demo

Youtube

← Metadata

19
Stars
2
Forks
Watchers

Owner

verified publisher icon appuv

Metadata

This repository provides Python scripts to generate simulated data and produce it into Kafka topics, facilitating testing and development of Kafka-based applications and pipelines.

Back