js-renderer

JS renderer is an online puppeteer service to render pages with javascript (js). Mainly useful for web scraping (not using splash).

At times while scraping web pages you will come across websites or web pages that only render on a browser that renders the loaded javascript. If you curl it or use something like Scrapy, you just end up with not useful HTML.

This project aims to solve that issue with Puppeteer. With Scrapy you can use Splash but it is Scrapy specific and not easy to configure.

Uses

This project uses Puppeteer to render the page as a full browser and Express to open Puppeter as an API.

Run locally

If you have node install you can do:

npm install
npm start

If you want to run with docker, execute the following:

npm install
docker-compose up

The hit http://localhost:8080/api/render?url=https://instagram.com

How to use it

If you want to use it for scraping, use the following URL on Vercel:

https://js-renderer.now.sh/api/render?url=https://instagram.com

Styles broken

Styles and images will look broken but the HTML tags will be there. Happy Web Scraping!