no negative values

[atwahsz]

greetings. i've been trying to use your DTW library to shift two series (DEPTH in this case) to each other .

a value is generated howerver . it's always postive . and in my case i need it to specifiy is the shift up (negative) or down (postive)

for each barrel

https://github.com/sudomaze/core-dtw

please guide me to the correct way

[johannfaouzi]

Hi,

First, you must use the Sakoe-Chiba method (method='sakoechiba') to compute DTW with the Sakoe-Chiba band. This method has a window_size option, but the current implementation makes that it cannot be negative because it is assumed that the possible shifts are symmetric. See this example for an illustration.

However, you can provide your own region by using the method='region' method and providing the region parameter. See the documentation of this method.

Do you want one of these? Light orange corresponds to the band, and dark yellow corresponds to the optimal path.

Source code to generate this image:

import matplotlib.pyplot as plt
import numpy as np
from pyts.metrics import dtw_region


# Create two time series with 24 points
n_timestamps = 24
rng = np.random.RandomState(42)
x, y = rng.randn(2, n_timestamps)

# Define the shift and create the regions
shift = 8
region_pos_shift = np.array([np.arange(24), np.clip(np.arange(24) + shift, 1, 24)])
region_neg_shift = np.array([np.clip(np.arange(24) - shift, 0, 24), np.arange(24) + 1])

# Plot the results
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))

# Positive shift
dtw_pos_shift, path_pos_shift = dtw_region(x, y, region=region_pos_shift, return_path=True)

mask_pos_shift = np.zeros((n_timestamps, n_timestamps))
for i, (j, k) in enumerate(region_pos_shift.T):
    mask_pos_shift[j:k, i] = 0.5
for i, j in path_pos_shift.T:
    mask_pos_shift[j, i] = 1.

ax1.imshow(mask_pos_shift, origin='lower', cmap='Wistia', vmin=0, vmax=1)
ax1.set_xticks(np.arange(-.5, n_timestamps, 1), minor=True)
ax1.set_yticks(np.arange(-.5, n_timestamps, 1), minor=True)
ax1.grid(which='minor', color='b', linestyle='--', linewidth=1)
ax1.set_xticks(np.arange(0, n_timestamps, 4))
ax1.set_yticks(np.arange(0, n_timestamps, 4))
ax1.set_title('Positive shift: DTW = {:.3f}'.format(dtw_pos_shift),
              fontsize=18)


# Negative shift
dtw_neg_shift, path_neg_shift = dtw_region(x, y, region=region_neg_shift, return_path=True)

mask_neg_shift = np.zeros((n_timestamps, n_timestamps))
for i, (j, k) in enumerate(region_neg_shift.T):
    mask_neg_shift[j:k, i] = 0.5
for i, j in path_neg_shift.T:
    mask_neg_shift[j, i] = 1.

ax2.imshow(mask_neg_shift, origin='lower', cmap='Wistia', vmin=0, vmax=1)
ax2.set_xticks(np.arange(-.5, n_timestamps, 1), minor=True)
ax2.set_yticks(np.arange(-.5, n_timestamps, 1), minor=True)
ax2.grid(which='minor', color='b', linestyle='--', linewidth=1)
ax2.set_xticks(np.arange(0, n_timestamps, 4))
ax2.set_yticks(np.arange(0, n_timestamps, 4))
ax2.set_title('Negative shift: DTW = {:.3f}'.format(dtw_neg_shift),
              fontsize=18);

Hope this helps you a bit and sorry for the delay.

[atwahsz]

how can i force it to ouput negative value ? so i can apply the change to the dataset and display it just like the github notebook i sent it has all the data

[atwahsz]

we want DTW to output either postive or negative shift in order to put PHIN in the right place in depth

[johannfaouzi]

DTW can only output a non-negative value: the minimum value is 0, when there exists a path such that the values perfectly match in both time series. DTW is a distance-like metric and measures similarity between two time series: the lower, the more similar the time series.

[atwahsz]

in our case we want to use it for Depth series , where shallow is negative . deeper is postive

and we are matching the PHIN values the segemnted series to the full continous one

any advice ? or guidance of the right approach ?

[atwahsz]

negative(in terms of shift)

[atwahsz]

the target here is the shifted series with the new depths

[atwahsz]

[johannfaouzi]

I don't understand. The matching on these images looks pretty good. Do you want to do that for your data?

[atwahsz]

Yes .

[johannfaouzi]

You don't need dynamic time warping if there is no compression or dilation, just compute the R2 scores for each shift and find the index of the maximum:

import matplotlib.pyplot as plt
import numpy as np
from sklearn.metrics import r2_score

# Generate a toy dataset
x_size, y_size = 100, 60
rng = np.random.RandomState(42)
x = np.cumsum(rng.randn(x_size))
y = x[20:80] + rng.randn(y_size) / 2

# Find the optimal shift
r2_scores = []
for i in range(x.size - y.size):
    r2_scores.append(r2_score(y, x[i:i + y.size]))
r2_scores = np.array(r2_scores)
idxmax = r2_scores.argmax()

plt.figure(figsize=(12, 4))

plt.subplot(1, 2, 1)
plt.plot(x, label='x')
plt.plot(np.arange(30, 90), y, label='y')
plt.title('Before matching: R2 = {:.3f}'.format(r2_score(x[30:90], y)))
plt.legend()

plt.subplot(1, 2, 2)
plt.plot(x, label='x')
plt.plot(np.arange(idxmax, idxmax + y_size), y, label='y (matched)')
plt.title('After matching: R2 = {:.3f}'.format(r2_score(x[idxmax:idxmax + y_size], y)))
plt.legend();

[atwahsz]

Is it possible to limit the shift by a window?

[johannfaouzi]

Of course. In this example I tried out all the possibles shifts, but you can limit them to a window. Just change the iterator in the for loop.