pyts-repro: a repository to compare the accuracies between the results published in the literature and the accuracies using pyts

Introduction

pyts (GitHub, Documentation, PyPI) is a Python package dedicated to time series classification. It provides several tools to preprocess the data and implementations for many state-of-the-art algorithms. The goal of this repository is to compare the accuracies between the ones published in the literature and the ones using pyts. Alongside the high code coverage, it aims at ensuring that the implementations in pyts are reliable.

Note: Most algorithms have hyperparameters that need to be fine-tuned for each dataset. If the values of these hyperparamaters are not directly available, a gridsearch is performed using the testing set. For each of those algorithms, the accuracy reported in the pyts column is the minimum of the accuracy reported in the article and the highest accuracy obtained with the gridsearch (to avoid any overestimation of the performance of the algorithm because of data leakage). The same gridsearches as the ones presented in the articles are usually not done for computational reasons.

Datasets

The datasets used are taken from the UCR Time Series Classification Archive. On this website, you can download the datasets (a password is required to unzip the file, you can find it by reading the PDF or the PowerPoint) and the table with the results using a 1NN classifier with several metrics (Euclidean Distance, Dynamic Time Warping and Dynamic Time Warping with a learning warping window). For computational reasons, the algorithms are only tested on the smallest datasets. This way, anyone can run the notebooks by themselves on a single machine and verify the results. The selected datasets are presented in the table below.

The notebooks are self-contained as there is some code to download the datasets used in this benchmark. Make sure to have an Internet connection when you run them!

Comparisons

Link to the notebook

1NN classifier with Euclidean Distance, Dynamic Time Warping and Dynamic Time Warping with a learned warping window

Bag-of-Patterns transformer followed by a 1NN classifier using Euclidean distance

Link to the notebook

BOSS transformer followed by a 1NN classifier using the BOSS metric

Link to the notebook

BOSSVS classifier

Link to the notebook

Learning-Shapelet classifier

Link to the notebook

ROCKET transformer followed by a Ridge Classifier with built-in cross-validation

Link to the notebook

SAXVSM classifier

Link to the notebook

ShapeletTransform transformer followed by a Support Vector Machine with a linear kernel

Link to the notebook

TimeSeriesForest classifier

Link to the notebook

TSBF classifier

Link to the notebook

WEASEL transformer followed by a logistic regression

Link to the notebook