How to upsample during the decoding stage

[SC-shendazt]

Hi Dr.HuguesTHOMAS,I have a problem that has been bothering me for quite a long time, how to upsample in the decoding stage, as shown by the red line in the figure, I want to upsample the output of Unary block to the feature dimension of N1*64. Do I need to make changes in the architecture?

[HuguesTHOMAS]

If I understand correctly, you want to concatenate the features from all layers as a final feature, instead of only using the N1 x 64 last feature, right?

In that case, using the closest pool function is the right way to go. If you don't want to modify too much of the code, you can only use it from one layer to another, as the indices for the closest_pool are computed in advance. So, for example, to upsample N3 x 256, you can do:

x = closest_pool(closest_pool(x, batch.upsamples[layer - 1]), batch.upsamples[layer - 2])  # Layer should be equal to 2 here

Or, for a solution that you can apply to any layer:

for up_l in range(layer -1, -1, -1):  # loop from (layer-1) to 0 included
    x = closest_pool(x, batch.upsamples[up_l]) 

You can do that at the block you want to get the features from and just append these in a list that you can handle after the decoder loop.

[SC-shendazt]

If I understand correctly, you want to concatenate the features from all layers as a final feature, instead of only using the N1 x 64 last feature, right?

In that case, using the closest pool function is the right way to go. If you don't want to modify too much of the code, you can only use it from one layer to another, as the indices for the closest_pool are computed in advance. So, for example, to upsample N3 x 256, you can do:

x = closest_pool(closest_pool(x, batch.upsamples[layer - 1]), batch.upsamples[layer - 2])  # Layer should be equal to 2 here

Or, for a solution that you can apply to any layer:

for up_l in range(layer -1, -1, -1):  # loop from (layer-1) to 0 included
    x = closest_pool(x, batch.upsamples[up_l]) 

You can do that at the block you want to get the features from and just append these in a list that you can handle after the decoder loop.

thanks very much！！

[SC-shendazt]

Dr.HuguesTHOMAS, Have you calculated the number of floating point calculations per second?（FLOPS）

[HuguesTHOMAS]

No I have not