Implement successor of Savitzky-Golay: The Modified Sinc Smoother

[MothNik]

🚶➡️🏃 Proposed Enhancement

In 2023, some researchers put the Savitzky-Golay-filter to the test and figured out that - despite its wide-spread use - it is actually far from being a good smoother. Their publication Why and How Savitzky–Golay Filters Should Be Replaced introduces a new smoother called "Modified Sinc Smoother" which cuts short all the disadvantages of the Savitzky-Golay-filter - first and foremost that it does many things but fully removing noise. The publication really goes deep into the topic and explains the reasons for this quite nicely.

Given this, an implementation of the Modified Sinc Smoother would be a nice addition to chemotools. It would be light-weight smoother (not heavy like WhittakerSmooth) with yet excellent smoothing capabilities. Fun fact: the Savitzky-Golay and Modified Sinc Smoother share the same parameters (window_length and polyorder; the publication states that polyorder is the wrong naming convention and should be poly_degree), so the Modified Sinc Smoother could become a simple drop-in-replacement for the Savitzky-Golay.

🧑‍💻 Implementation aspects

Both the Savitzky-Golay- and the Modified Sinc Smoother belong to the class of so-called Finite Impulse Response Filters. Their main principle is to convolve the signal with a filter. The only thing they differ in is the shape of the filter, so basically both smoothers can share the same base class. Then, they only provide the computation of their filter coefficients which keeps the transformer classes themselves lean while reusing a lot of logic that then only has to be tested once. All that's required is a common interface where the individual filters can hand over their respective filter coefficients as depicted below.

[paucablop]

Hi @MothNik

Excellent idea, this is exactly what I also wanted chemotools to be, a place where new developments can be implemented!! After going through the paper, I agree with creating the FIR interface, it is nice for modularization and reusability of the code! I will make a new branch with this one!! :smile: :muscle:

[MothNik]

@paucablop Just wanted to mention that this issue could become obsolete if this scipy issue would be implemented. The authors of the article linked in the initial issue description propose 2 alternatives to the Savitzky-Golay filter:

• the Modified Sinc Smoother
• the weighted Savitzky-Golay filter

While the former is slightly superior, I found it to suffer from quite a few drawbacks. The most important of them is that there is no simple analytical form for the derivative computation (the first derivative is possible, but has already quite a complicated expression, and numerical accuracy becomes an issue for higher order derivatives).

The weighted Savitzky-Golay filter, on the other hand, only requires computing the derivatives of a polynomial, which is a no-brainer with a simple analytical form and numerical stability. If the weights are properly chosen, it easily keeps up with the Modified Sinc Smoother.

Btw, if you want to support large filters with high polynomial orders for the Savitzky-Golay filter, this scipy issue might be relevant.

[MothNik]

Wow, great to see this made it into chemotools! 🤩 Nice job! 🤩