matrixprofile-ts icon indicating copy to clipboard operation
matrixprofile-ts copied to clipboard

Published 20 hours ago • verified publisher icon target

Handling constant subsequence

Open oentaryorj opened this issue 6 years ago • 7 comments

In some occasion, a time series subsequence can be flat/constant for a while, which may lead to a zero standard deviation for that subsequence. However, the the current codes does not seem to handle this case yet. Is there a way to fix this?

oentaryorj avatar Dec 13 '18 05:12 oentaryorj

@oentaryorj This is a great call-out. There are a couple of options:

  1. We could remove flat/constant segments as a pre-processing routine in the library

  2. We could report these segments back as motifs (though admittedly this is a "trivial" motif that may not be meaningful).

Since it sounds like you've encountered this situation in practice, what are your thoughts?

vanbenschoten avatar Dec 13 '18 18:12 vanbenschoten

@oentaryorj Something that's worked for me was to add some very small noise to the signal. This has the added benefit that the flat regions can be detected later on as different segments of the time series.

aouyang1 avatar Dec 29 '18 00:12 aouyang1

@vanbenschoten: Sorry for the late response - just back from vacation :) I would be more inclined toward option 2, although we could probably put an option to ignore/skip the flat segments.

@aouyang1: Thanks for the suggestion. Do you need to modify the current codes to make this work? Any other numerical issues encountered?

oentaryorj avatar Jan 02 '19 06:01 oentaryorj

If you read the original SAX paper (SAX was invented by the same people who invented the Matrix Profile) you will see that they handle subsequences with very low standard deviation (that includes the constant subsequence, as that has standard deviation zero) by not dividing by the standard deviation when normalizing. (The threshold for the standard deviation is a parameter here.) I think incorporating that into the matrix profile makes the most sense.

ameya98 avatar Jun 18 '19 00:06 ameya98

Agreed. Found this from stack overflow as well https://github.com/scikit-learn/scikit-learn/blob/7389dbac82d362f296dc2746f10e43ffa1615660/sklearn/preprocessing/data.py#L70 treating the standard deviation as 1 keeping the existing subsequence values and not normalizing.

Since we're trying to compare to subsequences together such that one subsequence's spectral power does not bias over another, would it make sense to divide the signal such that we satisfy parseval's theorem?

aouyang1 avatar Jun 18 '19 02:06 aouyang1

Thanks for the inputs. So I guess "ignoring" the standard deviation means option 2 in @vanbenschoten's reply (i.e., treating constant subsequence as "trivial" motif)? This makes sense to me, although it may depend on the application context.

oentaryorj avatar Jun 18 '19 02:06 oentaryorj

Performing the mean subtraction but not the division by the standard deviation in the distance profile computation should suffice as well?

ameya98 avatar Jun 18 '19 03:06 ameya98