skimpy
skimpy copied to clipboard
aeturrell
skimpy is a light weight tool that provides summary statistics about variables in data frames within the console.
Skimpy
A light weight tool for creating summary statistics from dataframes.
skimpy is a light weight tool that provides
summary statistics about variables in data frames within the console or your interactive Python window.
Think of it as a super-charged version of df.describe().
You can find the documentation here.
Quickstart
skim a dataframe and produce summary statistics within the console using:
from skimpy import skim
skim(df)
where df is a dataframe.
If you need to a dataset to try skimpy out on, you can use the built-in test dataframe:
#| output: asis
from skimpy import skim, generate_test_data
df = generate_test_data()
skim(df)
╭───────────────────────────────────── skimpy summary ──────────────────────────────────────╮ │ Data Summary Data Types Categories │ │ ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓ ┏━━━━━━━━━━━━━┳━━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━━━━━━┓ │ │ ┃ dataframe ┃ Values ┃ ┃ Column Type ┃ Count ┃ ┃ Categorical Variables ┃ │ │ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩ ┡━━━━━━━━━━━━━╇━━━━━━━┩ ┡━━━━━━━━━━━━━━━━━━━━━━━┩ │ │ │ Number of rows │ 1000 │ │ float64 │ 3 │ │ class │ │ │ │ Number of columns │ 10 │ │ category │ 2 │ │ location │ │ │ └───────────────────┴────────┘ │ datetime64 │ 2 │ └───────────────────────┘ │ │ │ int64 │ 1 │ │ │ │ bool │ 1 │ │ │ │ string │ 1 │ │ │ └─────────────┴───────┘ │ │ number │ │ ┏━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━┳━━━━━━━━━┳━━━━━━━┳━━━━━━┳━━━━━━┳━━━━━━━━┓ │ │ ┃ column_name ┃ NA ┃ NA % ┃ mean ┃ sd ┃ p0 ┃ p25 ┃ p75 ┃ p100 ┃ hist ┃ │ │ ┡━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━╇━━━━━━━━━╇━━━━━━━╇━━━━━━╇━━━━━━╇━━━━━━━━┩ │ │ │ length │ 0 │ 0 │ 0.5 │ 0.36 │ 1.6e-06 │ 0.13 │ 0.86 │ 1 │ █▃▃▃▄█ │ │ │ │ width │ 0 │ 0 │ 2 │ 1.9 │ 0.0021 │ 0.6 │ 3 │ 14 │ █▃▁ │ │ │ │ depth │ 0 │ 0 │ 10 │ 3.2 │ 2 │ 8 │ 12 │ 20 │ ▁▄█▆▃▁ │ │ │ │ rnd │ 120 │ 12 │ -0.02 │ 1 │ -2.8 │ -0.74 │ 0.66 │ 3.7 │ ▁▄█▅▁ │ │ │ └──────────────┴──────┴───────┴────────┴──────┴─────────┴───────┴──────┴──────┴────────┘ │ │ category │ │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓ │ │ ┃ column_name ┃ NA ┃ NA % ┃ ordered ┃ unique ┃ │ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩ │ │ │ class │ 0 │ 0 │ False │ 2 │ │ │ │ location │ 1 │ 0.1 │ False │ 5 │ │ │ └───────────────────────────┴─────────┴────────────┴──────────────────┴────────────────┘ │ │ datetime │ │ ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓ │ │ ┃ column_name ┃ NA ┃ NA % ┃ first ┃ last ┃ frequency ┃ │ │ ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩ │ │ │ date │ 0 │ 0 │ 2018-01-31 │ 2101-04-30 │ M │ │ │ │ date_no_freq │ 3 │ 0.3 │ 1992-01-05 │ 2023-03-04 │ None │ │ │ └────────────────────┴──────┴─────────┴────────────────┴────────────────┴──────────────┘ │ │ string │ │ ┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓ │ │ ┃ column_name ┃ NA ┃ NA % ┃ words per row ┃ total words ┃ │ │ ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩ │ │ │ text │ 6 │ 0.6 │ 5.8 │ 5800 │ │ │ └─────────────────────┴───────┴──────────┴────────────────────────┴────────────────────┘ │ │ bool │ │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ │ │ ┃ column_name ┃ true ┃ true rate ┃ hist ┃ │ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ │ │ booly_col │ 520 │ 0.52 │ █ █ │ │ │ └─────────────────────────────┴─────────────┴────────────────────────┴─────────────────┘ │ ╰─────────────────────────────────────────── End ───────────────────────────────────────────╯
It is recommended that you set your datatypes before using skimpy (for example converting any text columns to pandas string datatype), as this will produce richer statistical summaries. However, the skim function will try and guess what the datatypes of your columns are.
skimpy also comes with a clean_columns function as a convenience. This slugifies column names. For example,
import pandas as pd
from rich import print
from skimpy import clean_columns
columns = [
"bs lncs;n edbn ",
"Nín hǎo. Wǒ shì zhōng guó rén",
"___This is a test___",
"ÜBER Über German Umlaut",
]
messy_df = pd.DataFrame(columns=columns, index=[0], data=[range(len(columns))])
print("Column names:")
print(list(messy_df.columns))
Column names:
[ 'bs lncs;n edbn ', 'Nín hǎo. Wǒ shì zhōng guó rén', '___This is a test___', 'ÜBER Über German Umlaut' ]
Now let's clean these—by default what we get back is in snake case:
clean_df = clean_columns(messy_df)
print(list(clean_df.columns))
4 column names have been cleaned
[ 'bs_lncs_n_edbn', 'nin_hao_wo_shi_zhong_guo_ren', 'this_is_a_test', 'uber_uber_german_umlaut' ]
Other naming conventions are available, for example camel case:
clean_df = clean_columns(messy_df, case="camel")
print(list(clean_df.columns))
4 column names have been cleaned
['bsLncsNEdbn', 'ninHaoWoShiZhongGuoRen', 'thisIsATest', 'uberUberGermanUmlaut']
Requirements
You can find a full list of requirements in the pyproject.toml file. The main requirements are:
python >=3.8,<4.0.0
click ^8.1.3
rich >=10.9,<13.0
pandas ^1.3.2
Pygments ^2.10.0
typeguard ^2.12.1
jupyter ^1.0.0
ipykernel ^6.7.0
numpy ^1.22.2
black ^22.3.0
You can try this package out right now in your browser using this Google Colab notebook (requires a Google account). Note that the Google Colab notebook uses the latest package released on PyPI (rather than the development release).
Installation
You can install the latest release of skimpy via pip from PyPI:
$ pip install skimpy
To install the development version from git, use:
$ pip install git+https://github.com/aeturrell/skimpy.git
For development, see the Contributor Guide.
Usage
This package is mostly designed to be used within an interactive console session or Jupyter notebook
from skimpy import skim
skim(df)
However, you can also use it on the command line:
$ skimpy file.csv
Features
- Support for boolean, numeric, datetime, string, and category datatypes
- Command line interface in addition to interactive console functionality
- Light weight, with results printed to terminal using the rich package.
- Support for different colours for different types of output
- Rounds numerical output to 2 significant figures
skim accepts keyword arguments that change the colour of the top level column headers. For example, to change the colour to magenta, it's
skim(df, header_style="italic magenta")
╭───────────────────────────────────── skimpy summary ──────────────────────────────────────╮ │ Data Summary Data Types Categories │ │ ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓ ┏━━━━━━━━━━━━━┳━━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━━━━━━┓ │ │ ┃ dataframe ┃ Values ┃ ┃ Column Type ┃ Count ┃ ┃ Categorical Variables ┃ │ │ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩ ┡━━━━━━━━━━━━━╇━━━━━━━┩ ┡━━━━━━━━━━━━━━━━━━━━━━━┩ │ │ │ Number of rows │ 1000 │ │ float64 │ 3 │ │ class │ │ │ │ Number of columns │ 10 │ │ category │ 2 │ │ location │ │ │ └───────────────────┴────────┘ │ datetime64 │ 2 │ └───────────────────────┘ │ │ │ int64 │ 1 │ │ │ │ bool │ 1 │ │ │ │ string │ 1 │ │ │ └─────────────┴───────┘ │ │ number │ │ ┏━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━┳━━━━━━━━━┳━━━━━━━┳━━━━━━┳━━━━━━┳━━━━━━━━┓ │ │ ┃ column_name ┃ NA ┃ NA % ┃ mean ┃ sd ┃ p0 ┃ p25 ┃ p75 ┃ p100 ┃ hist ┃ │ │ ┡━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━╇━━━━━━━━━╇━━━━━━━╇━━━━━━╇━━━━━━╇━━━━━━━━┩ │ │ │ length │ 0 │ 0 │ 0.5 │ 0.36 │ 1.6e-06 │ 0.13 │ 0.86 │ 1 │ █▃▃▃▄█ │ │ │ │ width │ 0 │ 0 │ 2 │ 1.9 │ 0.0021 │ 0.6 │ 3 │ 14 │ █▃▁ │ │ │ │ depth │ 0 │ 0 │ 10 │ 3.2 │ 2 │ 8 │ 12 │ 20 │ ▁▄█▆▃▁ │ │ │ │ rnd │ 120 │ 12 │ -0.02 │ 1 │ -2.8 │ -0.74 │ 0.66 │ 3.7 │ ▁▄█▅▁ │ │ │ └──────────────┴──────┴───────┴────────┴──────┴─────────┴───────┴──────┴──────┴────────┘ │ │ category │ │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓ │ │ ┃ column_name ┃ NA ┃ NA % ┃ ordered ┃ unique ┃ │ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩ │ │ │ class │ 0 │ 0 │ False │ 2 │ │ │ │ location │ 1 │ 0.1 │ False │ 5 │ │ │ └───────────────────────────┴─────────┴────────────┴──────────────────┴────────────────┘ │ │ datetime │ │ ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓ │ │ ┃ column_name ┃ NA ┃ NA % ┃ first ┃ last ┃ frequency ┃ │ │ ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩ │ │ │ date │ 0 │ 0 │ 2018-01-31 │ 2101-04-30 │ M │ │ │ │ date_no_freq │ 3 │ 0.3 │ 1992-01-05 │ 2023-03-04 │ None │ │ │ └────────────────────┴──────┴─────────┴────────────────┴────────────────┴──────────────┘ │ │ string │ │ ┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓ │ │ ┃ column_name ┃ NA ┃ NA % ┃ words per row ┃ total words ┃ │ │ ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩ │ │ │ text │ 6 │ 0.6 │ 5.8 │ 5800 │ │ │ └─────────────────────┴───────┴──────────┴────────────────────────┴────────────────────┘ │ │ bool │ │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ │ │ ┃ column_name ┃ true ┃ true rate ┃ hist ┃ │ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ │ │ booly_col │ 520 │ 0.52 │ █ █ │ │ │ └─────────────────────────────┴─────────────┴────────────────────────┴─────────────────┘ │ ╰─────────────────────────────────────────── End ───────────────────────────────────────────╯
Contributing
Contributions are very welcome. To learn more, see the Contributor Guide.
Note that you will need Quarto and Make installed to build the docs. You can preview the docs using poetry run quarto preview --execute. You can build them with make.
License
Distributed under the terms of the MIT license, skimpy is free and open source software. You can find the license here
Issues
If you encounter any problems, please file an issue along with a detailed description.
Credits
This project was generated from @cjolowicz's Hypermodern Python Cookiecutter template.
skimpy was inspired by the R package skimr and by exploratory Python packages including pandas_profiling and dataprep, from which the clean_columns function comes.
The package is built with poetry, while the documentation is built with Quarto. Tests are run with nox.
aeturrell