Micro-website-api

An API microservice that crawls dynamic website powered by puppeteer.

How To Use

a. Deploy to Now.sh

One-click deploy to △ now

b. Docker image

$ docker build -t micro-website-api .
$ docker run --rm -it \
  -p 3000:3000 \
  -e "TIMEOUT=300000" \
  micro-website-api

API

Environment variables

Demo

Note: You should deploy your own service for production usage.

1. Prerender a SPA. [WebUI] [API]

2. With target selector. [WebUI] [API]

3. Click to send an ajax and wait for a selector. [WebUI] [API]

4. Click to send an ajax and wait for a long timeout. [WebUI] [API]

5. Respond JSON format. [WebUI] [API]

6. Multiple actions. [WebUI] [API]

Developer Guide

Requirements

• node >= 9.1.0
• npm >= 5.5.1
• yarn >= 1.3.2

$ git clone https://github.com/evenchange4/micro-website-api.git
$ yarn install --pure-lockfile
$ yarn run dev # dev server
$ yarn start   # prod server

Test

$ yarn run format
$ yarn run eslint
$ yarn run test:watch
$ yarn run flow

Inspiration

• https://wrapapi.com/
• https://github.com/ebidel/try-puppeteer
• https://github.com/zenato/puppeteer-renderer

CONTRIBUTING

• ⇄ Pull requests and ★ Stars are always welcome.
• For bugs and feature requests, please create an issue.
• Pull requests must be accompanied by passing automated tests ($ yarn run test).

CHANGELOG

LICENSE

MIT: http://michaelhsu.mit-license.org