Deeper, a data enrichment system through progressive deep web crawling.

Deeperlib is a library for data enrichment with web data. Given a local data table, it is able to efficiently find the matching records in a deep website through keyword search interface API, so that the returned data can be used to enrich the local data.

APIs Supportted

The current version implemented API for the following websites as examples:

• DBLP（DataBase systems and Logic Programming）
• YELP（Yelp Fusion API）
• AMiner（arnetminer）

Custom

You can also implement other APIs based on your needs by implementing a subclass of deeper.api.simapi and pass it to deeper.core.smartcrawl.

Documentation

Fantastic documentation is available at https://sfu-db.github.io/deeperlib/

Requirements

• pqdict>=1.0.0
• requests>=2.18.4
• simplejson>=3.11.1
• rauth>=0.7.3

Requests officially supports Python 2.7.13, and runs great on PyPy.

Installation and Update

pip install deeperlib

pip install --upgrade deeperlib

Publication

[1]. P. Wang et al. Progressive Deep Web Crawling Through Keyword Queries For Data Enrichment. SIGMOD 2019.
[2]. P. Wang et al. Deeper: A Data Enrichment System Powered by Deep Web. SIGMOD 2018 (demo).