Building a real-time analytics dashboard with Streamlit, Apache Pinot, and Apache Kafka

Clone repository

[source, bash]

git clone [email protected]:mneedham/pinot-wiki.git && cd pinot-wiki

Spin up all components

[source, bash]

docker-compose up

or on the Mac M1:

[source, bash]

docker-compose -f docker-compose-m1.yml up

Setup Python

Ingest Wikipedia events

[source, bash]

python -m venv .venv source venv/bin/activate pip install -r requirements.txt

Create Kafka topic

[source, bash]

docker exec -it kafka-wiki kafka-topics.sh
--bootstrap-server localhost:9092
--partitions 5
--topic wiki-events
--create

Ingest Wikipedia events

[source, bash]

python wiki_to_kafka.py

Check Wikipedia events are ingesting

[source, bash]

docker exec -it kafka-wiki kafka-run-class.sh kafka.tools.GetOffsetShell
--broker-list localhost:9092
--topic wiki-events

[souce, bash]

kafkacat -C -b localhost:9092 -t wiki-events

Add Pinot Table

[source, bash]

docker exec -it pinot-controller-wiki bin/pinot-admin.sh AddTable
-tableConfigFile /config/table.json
-schemaFile /config/schema.json
-exec

Open the Pinot UI http://localhost:9000/

Run Streamlit app

[source, bash]

streamlit run streamlit/app.py