• dataviper
• Example
• Why?
• Data Sources
• APIs
  • profile
  • pivot
  • joinability
• Issues and TODOs

dataviper

dataviper is a SQL-based tool to get the basic data preparation done in easy way, with doing

• Create "Data Profile" report of a table
• One-hot encode for "Categorical Columns" and create a "one-hot" table
• Check "Joinability" between 2 tables
• // TODO: and more

Example

pip install dataviper

Your main.py will look like this

from dataviper import Client
from dataviper.source import MySQL

client = Client(source=MySQL({
    'host': 'localhost',
    'user': 'root',
    'password': 'xxxxxx',
    'database': 'your_database'
}))

with client.connect():
    profile = client.profile('Your_Table')
    profile.to_excel()

python3 main.py
# Then you will get 'profile_Your_Table.xlsx' 🤗

Why?

It's known that "Data Profiling" needs to be done with scanning all the rows in a table. If you try to do this naively by pandas or any libraries which internally use pandas, it's not avoidable to use bunch of memory of your local machine and freeze your work.

dataviper is a SQL-based Data Profiling tool, which simply and dynamically generates SQLs and lets the database machine do the annoying calculation.

With dataviper, you don't have to have massive local computer. All you need are a stable network and reachable SQL db.

Data Sources

You can choose your data source from

• [ ] SQL Server
  • [x] profile
  • [x] pivot
  • [x] joinability
  • [ ] histogram
• [ ] MySQL
  • [x] profile
  • [ ] pivot
  • [ ] joinability
  • [ ] histogram
• [ ] PostgreSQL
• [ ] CSV
  • [x] profile
  • [ ] pivot
  • [ ] joinability
  • [ ] histogram
• [ ] Excel

APIs

profile

Create "Data Profile" excel file of a specified table.

When you have Sales table like this

do

with client.connect() as conn:
    table_name = 'Sales'
    profile = client.profile(table_name, example_count=3)
    profile.to_excel()

then you will get profile_Sales.xlsx file with

pivot

Spread categorical columns to N binary columns.

When you have Sales table like above, do

with client.connect() as conn:
    table_name = 'Sales'
    key = 'id'
    categorical_columns = ['region', 'sales_type']
    profile = client.get_schema(table_name)
    client.pivot(profile, key, categorical_columns)

then you will get Sales_PIVOT_YYYYmmddHHMM table with

joinability

Count how much 2 tables can be joined.

When you have Sales table like above, and Reps table like this

do

with client.connect() as conn:
    report = client.joinability(on={'Sales': 'rep_id', 'Reps': 'id'})
    report.to_excel()

then you will get join_Sales_Reps.xlsx file with

Issues and TODOs

• https://github.com/otiai10/dataviper/issues