Various filter size

[user4325]

Hi, I'm trying to use keras-tcn code to replicate a tcn algorithm build on pytorch: https://github.com/olivesgatech/Estimation-of-acoustic-impedance-from-seismic-data-using-temporal-convolutional-network

I think I'm getting close, but run into an error during the process.

For each Temporal Block (i.e. each dilation value [1, 2, 4 ...] ), the number of filters used for the convolutions (in the pytorch algo) can change. On the other hand, keras-tcn can take an array for the number of filters, which must be the size of the dilation array ( dilation = [1,2,4,8] --> nb_filters=[4,5,5,6] for example).

It seems to work except if use_skip_connection = True. The shape of the array to concatenate in this case are not maching. The error happens in tcn.py , at line 308 : x = layers.add(self.skip_connections)

I'm unfortunately missing knowledge to find a fix ... If anyone has an idea, i'll be happy to try

[philipperemy]

@user4325 well, it somehow makes sense that the dimensions do not match because the filters are of different sizes. Adding skip_connections sums the outputs over many layers. We can always add 1x1 conv to match the outputs of all the layers? Or have dense layers? For example if I have filters=[10,20,30], we can probably consider the biggest filter size (30) and consider the transformations 10->30 and 20->30. That would result in additional weights. Let me know what would be the best way to address this. How do they do in Pytorch?

[ghost]

Hi, thanks for your answer! I found a way to build the architecture as I wanted to, based on that: https://www.programmersought.com/article/13674618779/

Also, I might be a bit confuse about what skip_connections is exactly doing. So here is a quick explanation of what I did -> my own translation of the pytorch code I want to replicate, based on the previous link, and maybe two elements of solutions.

Basically, this is a sequence to sequence model, with a given number of Residual Blocks (One per dilation). Each block having a given number of filters and kernel size.

After several Residual blocks (example: 4 if dilation=[1,2,4,8]), the output need to be back to the dimension of the output sequence. This is done with 2 Conv1DTranspose layers.

1. What I have in mind is adding the initial input sequence to the output of the residual blocks, just before the Conv1DTranspose layers. To do so, I just used the Concatenate layer from tf.keras.

2. A residual block is the succession of 2 Conv1D layers. Within a residual block, a variable called shortcut is used to concatenate the result from the 2 Conv1D layers with the input of the residual block. If the shapes of the input of the residual block and the output are different, they use a conv1D layer with the appropriate number of filters and kernel_size directly on the input before concatenation of both (add layer). I guess that's where you are thinking of the 1*1 conv layer?

I'm now not sure that my step 1. is necessary as the information might already been taken into account by the shortcut variable within a residual block.

I hope I'm not too confusing in my explanation!

[loySoGxj]

Looking at the above url, it looks like a mixture of non-causal TCN and TCN. What kind of problem (background) are you thinking about? At least it doesn't seem to apply to time series.

[lminer]

Perhaps a value error should be raised if use_skip_connections is true and nb_filters is a list with non-equal elements.

[philipperemy]

@lminer good idea.

[philipperemy]

@lminer done: https://github.com/philipperemy/keras-tcn/commit/07e85e3f6a1c836af942956eb299841588dac77d

[philipperemy]

I'll close this as I don't think we need to do any dev on that one. Re-open if I'm wrong.