Python-DevOps

Python-DevOps copied to clipboard

Python-Devops, gathers Python stack for DevOps, these are usually my basic templates use for my implementations, so, feel free to use it and evolve it! Everything is Docker!

Table of contents

• Basic Python
• Basic Backend
• Scaling-up Backend
• Big data piping
• Apache big data
• Unit test
• Stress test
• Miscellaneous

Basic Python

1. AutoPEP8
2. Graph function dependencies

Basic Backend

1. Flask
2. Flask with MongoDB
3. REST API Flask
4. Flask Redis PubSub
5. Flask Mysql with REST API
6. Flask Postgres with REST API
7. Flask Elasticsearch
8. Flask Logstash with Gunicorn
9. MLFlow with Nginx reversed proxy

Scaling Backend

1. Flask SocketIO with Redis
2. Multiple Flask with Nginx Loadbalancer
3. Multiple Flask SocketIO with Nginx Loadbalancer
4. RabbitMQ and multiple Celery with Flask
5. Flask + Gunicorn + HAproxy

Big data piping

1. Streaming Tweepy to Elasticsearch
2. Scheduled crawler using Luigi Spotify to Elasticsearch
3. Airflow to Elasticsearch

Apache big data

1. Flask with Hadoop
2. Flask with Kafka
3. Flask with Hadoop Hive
4. PySpark with Jupyter and Hadoop
5. Flink with Jupyter (not working)
6. Apache Storm with Redis

Unit test

1. Pytest

Stress test

1. Locust

Miscellaneous (huseinhouse collection)

1. Elasticsearch + Kibana
2. Elasticsearch + Cerebro
3. Jupyter notebook
4. Jupyterhub
5. Jupyterhub + Github Auth
6. Hadoop cluster + Luigi + Jupyter Notebook
7. Kafka cluster
8. Apache Storm

How-to Docker

Every folders contain .yml for docker-compose. You need to install Docker-Compose first.

To run,

compose/build

To safely close

compose/down

To remove all images

docker rmi $(docker images -q)

To remove none images

docker rmi $(docker images -f "dangling=true" -q)

To remove all containers

docker rm $(docker ps -aq)