HTMLSpitter

Lightweight Docker image with NodeJS server to spit out HTML from loaded JS using Puppeteer and Chrome

Medium story: HTML from the Javascript world

The program is written in NodeJS with Typescript, in the src directory.

Description

Runs a NodeJS server accepting HTTP requests with two URL parameters:

• url which is the URL to prerender into HTML
• wait which is the optional load event to wait for before stopping the prerendering. It can be:
  • load (wait for the load event)
  • domcontentloaded (wait for the DOMContentLoaded event)
  • networkidle0 (default, wait until there is no network connections for at least 500 ms)
  • networkidle2 (wait until there are less than 3 network connections for at least 500 ms)

For example:

http://localhost:8000/?url=https://github.com/qdm12/htmlspitter

• The server scales up Chromium instances if needed
• It limits the number of opened pages per instance to prevent one page crashing all the other pages
• It has a 1 hour cache for loaded HTML
• It has a queue system for requests once the maximum number of pages/chromium instances is reached
• Not compatible with other architectures than amd64 as Chrome-Beta is only built for amd64 for now and is required.

Usage

1. Run the container

   docker run -it --rm --init -p 8000:8000 qmcgaw/htmlspitter
   

   You can also use docker-compose.yml.

Environment variables

Troubleshooting

Chrome fails to launch

If you obtain the error:

{"error":"Error: Failed to launch chrome!\nFailed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted\n\n\nTROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md\n"}

Then you might need to use seccomp with the chrome.json file of this repository:

wget https://raw.githubusercontent.com/qdm12/htmlspitter/master/chrome.json
docker run -it --rm --init --security-opt seccomp=$(pwd)/chrome.json -p 8000:8000 qmcgaw/htmlspitter

Details

Program

• A built-in local memory cache holds HTML content obtained the last hour and is limited in the size of characters it contains.
• A built-in pool of Chromium instances creates and removes Chromium instances according to the server load.
• Each Chromium instance has a limited number of pages so that if one page crashes Chromium, not all page loads are lost.
• As Chromium caches content, each instance is destroyed and re-created once it reaches a certain number of page loads.

Docker

• chrome.json may be required depending on your host OS.
• The --init flag is added to prevent eventual zombie Chromium processes to exist when the container stops the main NodeJS program.
• A built in healthcheck is implemented by running node build/healthcheck.js against a running instance.

Performance considerations

• Chromium is written in C++ and multi threaded so it scales well with more CPU cores
• The NodeJS program should not be the bottleneck because all the work is done by Chromium
• The bottleneck will be CPU and especially RAM used by Chromium instance(s)
• You can scale up by having multiple machines running the program, behind a load balancer

Development

• Either use the Docker container development image with Visual Studio Code and the remote development extension
• Or install Node and NPM on your machine

# Install all dependencies
npm i
# Transcompile the Typescript code to Javascript and run build/main.js with
npm run start

Test it with, for example:

wget -qO- http://localhost:8000/?url=https://github.com/qdm12/htmlspitter

You can also:

• Run tests

  npm t
  

• Run the sever with hot reload (performs npm run start on each .ts change)

  npx nodemon
  

• Build Docker

  docker build -t qmcgaw/htmlspitter .
  

  You can also specify the branch of Google Chrome from beta (default), stable and unstable

  docker build -t qmcgaw/htmlspitter --build-arg GOOGLE_CHROME_BRANCH=unstable
  

• There are two environment variables you might find useful:

  • PORT to set the HTTP server listening port
  • CHROME_BIN which is the path to the Chrome binary or Puppeteer-bundled

TODOs

• [ ] Show Chrome version at start
• [ ] Fake user agents
• [ ] Prevent recursive calls to localhost
• [ ] Format JSON or raw HTML
• [ ] Limit Chromium instances in terms of RAM
• [ ] Compression Gzip
• [ ] Sync same URL with Redis (not getting twice the same URL)
• [ ] Sync Cache with Postgresql or Redis depending on size
• [ ] Limit data size in Postgresql according to time created
• [ ] Unit testing
• [ ] ReactJS GUI
• [ ] Static binary in Scratch Docker image

Credits

• Credits to jessfraz for chrome.json
• The Google Chrome team
• The Puppeteer developers

License

This repository is under an MIT license