mutad
mutad copied to clipboard
Realtime Data Processing and Search Engine Implementation.
Mutad
The name Mutad is a reverse spelling of datum.
Overview
An implementation of a real-time data platform/search engine based on various technology stack. The implementation is rather for my own learning purpose.
![](https://github.com/ognis1205/mutad/raw/master/img/heatmap.gif?raw=true)
The above gif image demonstrates a real-time visualization of the tweet geoparsing of Mutad. According to my proceeding research, the amount of tweets with geospatial metadata is quite a few, say 10 per 10,000 tweets, from the Hosebird data feed. Thus I took another strategy to extract geospatial data from the tweet. Mutad implements real-time geoparsing based on CLAVIN. Because of my poor implementation, the accuracy is not that high yet.
![](https://github.com/ognis1205/mutad/raw/master/img/timeline.gif?raw=true)
The above gif image demonstrates a list timeline displaying the latest Tweets ordered from newest to oldest with a specified search query.
![](https://github.com/ognis1205/mutad/raw/master/img/trend.gif?raw=true)
The above gif image demonstrates a trend chart displaying the top most trended hashtags and tweet counts per a minute in the last hour.
Architecture
![](https://github.com/ognis1205/mutad/raw/master/img/mutad.jpeg?raw=true)
This diagram illustrates the overall architecture of Mutad. The technology stack is as follows:
Getting Started
Launch services by following the instructions below:
- Set up ELK Environment
- Set up Kafka Cluster
- Set up Storm Cluster
- Build Storm TRIE library
- Deploy Storm Collectors Topology
- Deploy Storm Processors Topology
- Start Spring Boot Elasticsearch Backend
- Start React User Interface
Future Work
The project is still WIP. See Issues.