toml-bench

Which toml package to use in python?

See also: toml-lang and PEP 680

Report

Version

The verions of the packages tested in this report.

Dumping None value

How the package dumps None value in python

Literally <package>.dumps(None)

Dumping key-None pair

How the package dumps key-value pair with value None

Literally <package>.dumps({"key": None})

Dumping list with None value

How the package dumps a list with None value in it.

Literally <package>.dumps({"key": [1, 2, 3, None, 5]})

Loading None-like values

How the package loads None-like value in string

Literally <package>.loads('v1 = "null" v2 = "None"')

Dumping a heterogenous array

How the package dumps a python dictionary with a heterogenous array.

Literally <package>.dumps({"v": [1, 1.2, True, "string"]})

Loading a heterogenous array

How the package loads a toml string with a heterogenous array.

Literally <package>.loads('v = [1, 1.2, True, "string"]')

Dumping a nested array

How the package dumps a python dictionary with a nested array.

Literally <package>.dumps({"v": [[1], [1, 2]]})

Loading a nested array

How the package loads a toml string with a nested array.

Literally <package>.loads('v = [[1], [1, 2]]')

Dumping keeps order of keys?

Whether the package preserves the order of the keys while dumps a python dictionary.

Thus, whether <package>.dumps({"c": 1, "a": 2, "b": 3}) yields a string like c = 1\na = 2\nb = 3\n.

Loading keeps order of keys?

Whether the package preserves the order of the keys while loads a TOML string.

Thus, whether <package>.loads('c = 1\na = 2\nb = 3\n') yields a dictionary with keys in the order of ['c', 'a', 'b'].

Dumping unicode

How the package dumps Unicode in python

Literally, <package>.dumps({"你好": "世界"})

Loaded unicode

How the package loads a file with unicode.

The file was created by:

## Create a file with unicode content
with open(self.datafile, "w", encoding="utf-8") as f:
    f.write('"你好" = "世界"\n')

## Use `<package>.load()` to load the file
with open(self.datafile, "r", encoding="utf-8") as f:
    loaded = <package>.load(f)

Compliance with valid tests in toml-test

Test the compliance with the standard test suites for valid toml files here:

https://github.com/BurntSushi/toml-test/

The tests come up with a JSON counterpart that can be used to valid whether loading the toml file yields the same result as the JSON counterpart.

Compliance with invalid tests in toml-test

Test the compliance with the standard test suites for invalid toml files here:

https://github.com/BurntSushi/toml-test/

• Not OK: The toml file is parsed without error, but expected to fail.
• OK: All files are failed to parse, as expected. Showing the last parsing error.

Compliance with valid tests in python tomllib test data

Test the compliance with python tomllib test data (since python 3.11) for valid toml files here:

https://github.com/python/cpython/tree/3.11/Lib/test/test_tomllib/data/valid

The tests come up with a JSON counterpart that can be used to valid whether loading the toml file yields the same result as the JSON counterpart.

Compliance with invalid tests in python tomllib test data

Test the compliance with python tomllib test data (since python 3.11) for invalid toml files here:

https://github.com/python/cpython/tree/main/Lib/test/test_tomllib/data/invalid

• Not OK: The toml file is parsed without error, but expected to fail.
• OK: All files are failed to parse, as expected. Showing the last parsing error.

Running speed with data provided by pytomlpp

Test the speed of loading and dumping the loaded using data provided by pytomlpp

https://github.com/bobfang1992/pytomlpp/raw/master/benchmark/data.toml

Running speed with data provided by rtoml

Test the speed of loading and dumping the loaded using data provided by rtoml

https://github.com/samuelcolvin/rtoml/raw/main/tests/data.toml

Running speed with data provided by tomli

Test the speed of loading and dumping the loaded using data provided by tomli

https://github.com/hukkin/tomli/raw/master/benchmark/data.toml

Other reports

• Tests with toml-test v1.2.0
• Tests with toml-test v1.1.0
• Tests with toml-test v1.0.0
• Tests with python 3.11 (tomllib included)

Run your own report

Install

pip install -U toml-bench

Generate your own report

toml-bench

Use a different data directory than the default one

toml-bench --datadir /tmp/toml-bench

Write the report to a markdown file

toml-bench --report ./README.md

Test with a different version of compliance set (BurntSushi/toml-test)

toml-bench --comver 1.0.0

Use a different number of iterations in speed tests

toml-bench --iter 5000

Test with different versions of packages

git clone https://github.com/pwwang/toml-bench.git
cd toml-bench
# See https://python-poetry.org/docs/cli/#add
# for how to specify a version constraint
poetry add "tomli=2.0.0"
poetry update
poetry install
poetry run toml-bench