web_full_stack_application

web_full_stack_application copied to clipboard

Full Stack Web Application

Purpose

This is a demo project for showing how one full stack web application should be like.

Introduction

features:

• [X] scrawler get data into database
• [X] webserver push data to browser periodically

Demo

Architecture

+-----------+                 +------------+              +------------+               +--------------+
|           |      frontend   |            |              |            |      data     |              |
|   Browser +<----------------+  Vue Server|              |   RestAPI  +<--------------+    Scrapy    |
|           |                 |            |              |            |               |              |
+------+----+                 +------------+              +------+-----+               +--------------+
       ^                                                         |
       |                                                         |
       |                                                         |data
       |                                                         |
       |                                                         |
       |                                                         v
       |                     +-------------+              +------+------+
       |     data            |             |    data      |             |
       +---------------------+  Websocket  +<-------------+   MongoDB   |
                             |             |              |             |
                             +-------------+              +-------------+

Workflow

1. scratch quotes from scrapy offical demo site(quotes), save to database(MongoDB), and show them through web.
2. Front end is vue.js project, including WebSocket client(socketio) component.
3. Back end is websocket server, which query data form MongoDB through ORM mongoose every period(1s), pushing quotes data to front end.
4. data picker use scrapy to crawl quotes to save to MongoDB.

Technology Stack

Scrapy + webservice[restful] + websocket + VueJS + MongoDB

Install

./bin/install.sh

Run

# run all services
./bin/start.sh

# stop all services
./bin/stop.sh

# start scrapy
./bin/start_scrapy.sh

Reference

• [x] mongodb install