vnet.pytorch icon indicating copy to clipboard operation
vnet.pytorch copied to clipboard

Concatenation operation in InputTransition causing confusion and memory abuse

Open tongdaxu opened this issue 6 years ago • 1 comments

As described in #01, the Concatenation operation in InputTransition has not been fix. Note that this could cause confusion in training, as the data should be of size() [Batchsize, Channel, Xsize, Ysize, Zsize] and the output to softmax should be of size() [Batchsize, Classnum, Xsize, Ysize, Zsize].

But as the broadcasting between [Batchsize16, XChannel, Xsize, Ysize, Zsize], [Batchsize, XChannel16, Xsize, Ysize, Zsize] bring [Batchsize16, XChannel16, Xsize, Ysize, Zsize], all the following layers would have 16 times more batchsize.

Mathematically this could be offset by running a lot of epochs, but could also make device suffers from memory issue. And each batch is equivalent to 16 un-shuffled batch.

tongdaxu avatar Mar 10 '19 16:03 tongdaxu

Thanks #1 ! I was confused for this problem! Now, my thoughts is right!

manglu097 avatar Oct 21 '23 12:10 manglu097