High error on time-series data

[cvronao]

Thank you so much for making such readable code for beginners in convolutional neural networks like me. I am trying to get some results with this code using time-series, xyz acceleration data. As opposed to images which are 2D, 3-channel data, time-series are 1D, 3-channel data. I have run the cnn_mnist examples included in this code, and have achieved satisfactory results. However, with the scaled time-series data that I have, I couldn't get any decent results with the same parameters and layer configurations. I suppose convolution computation for 1D and 2D is entirely different, and that I need to apply internal modifications if I want this to work. Please see below for the code I used.

Do I need to make modifications on the matrix operations level (convolution)? Where is this part in the code?

Note: I am also a python beginner, but I can understand the basic hierarchy of the code.

Code:

#!/usr/bin/env python
# coding: utf-8

import time
import numpy as np
import sklearn.datasets
import nnet


def run():
    # Data (https://archive.ics.uci.edu/ml/datasets/Human+Activity+Recognition+Using+Smartphones)
    # imgs has shape (n_imgs, n_channels_in, img_h, img_w)
    # filters has shape (n_channels_in, n_channels_out, img_h, img_w)
    # convout has shape (n_imgs, n_channels_out, img_h, img_w)
    xtrain = np.genfromtxt('/home/sclab/UCI_HAR_Dataset/train/Inertial Signals/body_acc_x_trainScaled.csv', delimiter=',')
    ytrain = np.genfromtxt('/home/sclab/UCI_HAR_Dataset/train/Inertial Signals/body_acc_y_trainScaled.csv', delimiter=',')
    ztrain = np.genfromtxt('/home/sclab/UCI_HAR_Dataset/train/Inertial Signals/body_acc_z_trainScaled.csv', delimiter=',')
    traindata = np.concatenate((xtrain, ytrain, ztrain), axis = 1)
    X_train = np.reshape(traindata, (-1, 3, 1, 64))
    y_train = np.genfromtxt('/home/sclab/UCI_HAR_Dataset/train/Inertial Signals/y_train.txt', dtype=int)

    xtest = np.genfromtxt('/home/sclab/UCI_HAR_Dataset/test/Inertial Signals/body_acc_x_testScaled.csv', delimiter=',')
    ytest = np.genfromtxt('/home/sclab/UCI_HAR_Dataset/test/Inertial Signals/body_acc_y_testScaled.csv', delimiter=',')
    ztest = np.genfromtxt('/home/sclab/UCI_HAR_Dataset/test/Inertial Signals/body_acc_z_testScaled.csv', delimiter=',')
    testdata = np.concatenate((xtest, ytest, ztest), axis = 1)
    X_test = np.reshape(testdata, (-1, 3, 1, 64))
    y_test = np.genfromtxt('/home/sclab/UCI_HAR_Dataset/test/Inertial Signals/y_test.txt', dtype=int) 
    n_classes = np.unique(y_train).size

    # Setup convolutional neural network
    nn = nnet.NeuralNetwork(
        layers=[
            nnet.Conv(
                n_feats=12,
                filter_shape=(3, 3),
                strides=(1, 1),
                weight_scale=0.1,
                weight_decay=0.001,
            ),
            nnet.Activation('relu'),
            nnet.Pool(
                pool_shape=(2, 2),
                strides=(2, 2),
                mode='max',
            ),
            nnet.Conv(
                n_feats=600,
                filter_shape=(3, 3),
                strides=(1, 1),
                weight_scale=0.1,
                weight_decay=0.001,
            ),
            nnet.Activation('relu'),
            nnet.Flatten(),
            nnet.Linear(
                n_out=n_classes,
                weight_scale=0.1,
                weight_decay=0.02,
            ),
            nnet.LogRegression(),
        ],
    )

    # Train neural network
    t0 = time.time()
    nn.fit(X_train, y_train, learning_rate=0.05, max_iter=3, batch_size=32)
    t1 = time.time()
    print('Duration: %.1fs' % (t1-t0))

    # Evaluate on test data
    error = nn.error(X_test, y_test)
    print('Test error rate: %.4f' % error)


if __name__ == '__main__':
    run()

[andersbll]

@cherryvr: I'm terribly sorry about the lateness of my reply. Due to illness I have been unable to work.

Regarding your problem, two ideas spring to mind: 1) I have never tried the convolution operations on 1D data and there might be an error in my implementation. 2) Have you remembered to scale your input to a reasonable range (a clasic mistake and when dealing with neural nets)? You want a reasonable scale between your input and your weights, also, make sure that the activation function doesn't squash the signal.

Feel free to ignore this post as I realize that you have probably moved on since then! :)

[cvronao]

Oh, it's okay! I am actually super thankful for being able to receive a reply even. I was worrying that my question was just a stupid one. >< Thank you so much for taking the time to answer. :)) I was able to learn a lot from your code, and realized that I have to standardize the input vectors. I understood that I should tune the learning rate and set reasonable initial range of weights. (I was able to implement it using Pylearn2, mainly because of the GPU functionality.)

I'm curious, is it possible to do 1D convolution on your GPU-enabled code deeppy?

Once again, thank you so much for all the help! And hope you are all well now. :)

[andersbll]

Thank you! :)

I have only tried 2D convolutions with DeepPy. I guess faking 1D convolution layers (that is, 2D with one side being 1) should work. Let me know if you should run into any problems.