Error while training LSTM

[ririya]

I've been frequently getting this error while training the net. Can someone tell me what the problem might be?

Index exceeds matrix dimensions.

Error in varObj>@(C)full(C(:,r)) (line 45) v = cellfun(@(C) full(C(:,r)), obj.v, 'UniformOutput', false);

Error in varObj/getmb (line 45) v = cellfun(@(C) full(C(:,r)), obj.v, 'UniformOutput', false);

Error in nnCostFunctionLSTM (line 13) Y = varObj(nn.Y.getmb(r), nn.defs, nn.defs.TYPES.OUTPUT);

Error in testSumNumbersGenerator>@(nn,r,newRandGen)nnCostFunctionLSTM(nn,r,newRandGen)

Error in gradientDescentAdaDelta (line 69) [J, dJdW, dJdB] = feval(f, nn, r, true);

Error in testSumNumbersGenerator (line 137) nn = gradientDescentAdaDelta(costFunc, nn, defs, [], [], [], [], 'Training Entire Network');

[ririya]

I've been trying to train a network that predicts the sum of the past two numbers. However i keep running into this kind of problem. If i increase the length of the sequence the training completes, however i don't get the expected output when i try to rebuild the sequence... Am I doing something wrong? This should be a very simple sequence to train. Here is my code

`clear; close all force; addpath('../../nn_gui'); addpath('../../nn_core'); addpath('../../nn_core/cuda'); addpath('../../nn_core/mmx'); addpath('../../nn_core/Optimizers'); addpath('../../nn_core/Activations'); addpath('../../nn_core/Activations'); addpath('../../nn_core/Wrappers'); addpath('../../nn_core/ConvNet'); addpath('Text');

PRECISION = 'double';

 % definitions(PRECISION, useGPU, whichThreads, plotOn) 

defs = definitions(PRECISION, true, [1], true);

% % Load the Shakespeares training set % Nchars = 50; % How many characters long for each sequence % offset = 0; % How much to shift the labels from the input data % txtpath = 'shakespeare_subset.txt'; % [X, vmap] = streamText2mat(txtpath, Nchars, offset); % offset = 1; % [Y, ~] = streamText2mat(txtpath, Nchars, offset);

seqLen = 2; N = 100000; offset = 0;

Xlong = 1:2;

for i=1:N

Xlong(end+1) = Xlong(end) + Xlong(end-1); if abs(Xlong(end)) > 2 Xlong(end) = -Xlong(end); end

% Xlong(end+1) = Xlong(end) +1; % if (Xlong(end) > 10) % Xlong(end) =1; % end end

% x123 = findstr(Xlong, [0 -1]); % Xlong(x123(1):x123(1)+5) % length(x123)

[X, vmap] = getSparseMatrix(Xlong, seqLen, offset); offset = 1;
[Y, ~] = getSparseMatrix(Xlong, seqLen, offset); % end

[BinCount] = hist(Xlong,vmap);

input_size = size(X{1},1); output_size = input_size; T = numel(X); % Length of all time sequences

% Both X and Y must include T=0 and T=Tf+1 'boundary conditions' filled % with zeros for convenience X(2:end+1) = X(:); X{1} = 0*X{1}; X(end+1) = X(1);

Y(2:end+1) = Y(:); Y{1} = 0*Y{1}; Y(end+1) = Y(1);

%%%%%%%%%%%%%%%%%%%%% Fine tuning Parameters %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% params = struct(); params.maxIter = precision(3000,defs); params.momentum = precision(0.9,defs); params.maxnorm = precision(0,defs); params.lambda = precision(0,defs); params.alphaTau = precision(0.25_params.maxIter,defs); % alpha_i = alpha_tau/(tau+i) (see "A Stochastic Quasi-Newton Method for Online Convex Optimization", Eqn. 7) params.denoise = precision(0,defs); % set to 0 to disable params.dropout = precision(0.6,defs); % set to 1 to disable params.miniBatchSize = precision(50,defs); % set to zero to disable mini-batches params.tieWeights = false; params.T = T; params.Tos = 0; % This is the "offset" time before the cost starts accumulating for the LSTM output % The idea here is that the LSTM can be fed with inputs % for n time steps, and won't be penalized for predictions % until t>=Tos. This helps by giving the LSTM context. % Optimization routine parameters:
params.alpha = precision(.001,defs); % If this is non-zero, use this learning rate for the entire network params.rho = precision(0.95, defs); % AdaDelta hyperparameter (don't generally need to modify) params.eps = precision(1e-6, defs); % AdaDelta hyperparameter (don't generally need to modify) params.cg.N = 10; % Max CG iterations before reset params.cg.sigma0 = 0.01; % CG Secant step-method parameter params.cg.jmax = 10; % Maximum CG Secant iterations params.cg.eps = 1e-4; % Update threshold for CG params.cg.mbIters = 10; % How many CG iterations per minibatch?

%%%%%%%%%%%%%%%%%%%%%%%%% Layer Setup %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% layers.af{1} = []; layers.sz{1} = [input_size 1 1]; layers.typ{1} = defs.TYPES.INPUT;

layers.af{end+1} = tanh_af(defs, []); layers.sz{end+1} = [128 1 1]; layers.typ{end+1} = defs.TYPES.LSTM;

layers.af{end+1} = softmax(defs, defs.COSTS.CROSS_ENTROPY); layers.sz{end+1} = [output_size 1 1]; layers.typ{end+1} = defs.TYPES.FULLY_CONNECTED; %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

if defs.plotOn nnShow(423, layers, defs); end

% Process Y such that first time sequence is stripped off and replaced by % a null at the end. This will cause LSTM to predict next character in the % sequence. The final prediction for a sequence should be null (zero).

X = varObj(X,defs,defs.TYPES.INPUT); Y = varObj(Y,defs,defs.TYPES.OUTPUT);

modelName = 'modelSumNumbers13.mat'; if ~exist(modelName)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% TRAINING %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

nn = nnLayers(params, layers, X, Y, {}, {}, defs); nn.initWeightsBiases();

costFunc = @(nn,r,newRandGen) nnCostFunctionLSTM(nn,r,newRandGen); nn = gradientDescentAdaDelta(costFunc, nn, defs, [], [], [], [], 'Training Entire Network'); %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

save(modelName, 'nn','vmap');

else load(modelName);

end

% outputText = []; outputSeq = [];

%% Generate some text by 'sampling' from LSTM % T_samp_temp = 1/35; % "Temperature" of random sampling process (higher temperatures lead to more randomness) T_samp_temp = 1; T_samp = 5000; % Length of sequence to generate % seedText = 'ROMEO:'; % seedText = 'Rafael Iriya';

% seedMatrix = unidrnd(3,3,6); seqLen = 2; seedMatrix = Xlong(1:seqLen); outputSeq = seedMatrix;

% Initial text (to provide some context) % vsize = numel(vmap); vsize = size(vmap,2); % Xs = full(ascii2onehot(seedText, vmap)); Xs = full(vec2map(seedMatrix, vmap)); % Xs = [zeros(vsize,1) Xs]; % Xtmp = zeros(vsize,1,size(seedMatrix,2)+1); Xtmp = zeros(vsize,1,size(seedMatrix,2)); Xtmp(:,1,:) = Xs; Xs = Xtmp;

nn.disableCuda(); nn.A{1} = varObj(Xs, nn.defs); preallocateMemory(nn, 1, size(seedMatrix,2)+1); % % Load up the LSTM with the context % for t=2:size(seedMatrix,2)+1 % feedforwardLSTM(nn, 1, t, false, true); % [~,cout] = max(nn.A{end}.v(:,1,t)); % % outputText = [outputText vmap(cout)] % outputSeq = [outputSeq vmap(:,cout)]; % %fprintf('%s', vmap(cout)); % end

for t=1:size(seedMatrix,2)+100 feedforwardLSTM(nn, 1, seqLen, false, true); [value,cout] = max(nn.A{end}.v(:,1,seqLen));

 outputSeq = [outputSeq vmap(:,cout)];

tmp = zeros(vsize,1);
tmp(cout) = 1;

for tt = 1:seqLen-1
    nn.A{1}.v(:,1,tt) = nn.A{1}.v(:,1,tt+1);
end

 nn.A{1}.v(:,1,end) = tmp;         

end

% Start sampling characters by feeding output back into the input for the % next time step % for t=size(seedMatrix,2)+2:size(seedMatrix,2)+T_samp % % Generate a random sample from the softmax probability distribution % % First, adjust/scale the distribution by a "temperature" that controls % % how likely we are to pick the maximum likelihood prediction % % P_next_char = exp(1/T_samp_temp*nn.A{end}.v(:,1,t-1)); % % P_next_char = P_next_char./sum(P_next_char); % normalize distribution % % cin = randsample(vsize,1,true,P_next_char); % % [value,cin] = max(nn.A{end}.v(:,1,t-1)); % % % fprintf('%s', vmap(cin)); % % outputText = [outputText vmap(cin)] % outputSeq = [outputSeq vmap(:,cin)]; %
% % Plot the distribution over characters % %{ % figure(777); % plot(P_next_char); % set(gca, 'XTick',1:numel(P_next_char), 'XTickLabel',vmap) % waitforbuttonpress; % %} %
% % Feed back the output to the input % % Generate the input for the next time step % tmp = zeros(vsize,1); % tmp(cin) = 1; % nn.A{1}.v(:,1,t) = tmp; %
% % Step the RNN forward % feedforwardLSTM(nn, 1, t, false, true); %
% end % disp(''); `

`function [X, vmap] = getSparseMatrix(Xlong, seqLen, offset)

Xlong = Xlong(:,1+offset:end); N = size(Xlong,2);

m = floor(N/seqLen);

Xlong = Xlong(1:m*seqLen);

dimX = size(Xlong,1);    

Npad = size(Xlong,2);

vmap = unique(Xlong', 'rows')'; vocabSize = size(vmap,2);

Xmap = zeros(1,Npad);

% Map all ASCII values to the reduced set for i=1:vocabSize indx = find(all(bsxfun(@eq, Xlong', vmap(:,i)'), 2)); % [~,indx]=ismember(vmap(:,i)',Xlong','rows') Xmap(indx) = i; end

 Xmap = reshape(Xmap,seqLen,[]);

%
% Xmap= hankel(Xmap, 1:seqLen); % Xmap = Xmap';%

% xreal = vmap(Xmap);

Xca = cell(seqLen,1); I = speye(vocabSize);

I = speye(vocabSize);

for t=1:seqLen
    Xt = Xmap(t,:);
    Xca{t} = I(Xt(:),:)';

end

Xfull1 = full(Xca{1});
Xfull2 = full(Xca{2});

 X = Xca;
`

[ririya]

I realized the problem is the trained network is not taking the memory into account, if I enter [1 2] the output should be -3, but its giving me something else, and if I change it to [3 2], [-1,2] or anything else it gives me the same result, which means its only taking 2 into account and not what comes before it. How do i change the network for it to take the memory into account?