Machine-Learning-Collection icon indicating copy to clipboard operation
Machine-Learning-Collection copied to clipboard

Published 20 hours ago • verified publisher icon aladdinpersson

ITransformer division error

Open dsantiago opened this issue 4 years ago • 0 comments

Hey Aladin, i think the division should be by the head_dim, in the code its using the full embed dimension

https://github.com/aladdinpersson/Machine-Learning-Collection/blob/bd4f07f5edb8968cf6129623ec73ba6ba777c964/ML/Pytorch/more_advanced/transformer_from_scratch/transformer_from_scratch.py#L62

You are dividing for square root of 256 and the correct should be square root o 64, 8 in the end

dsantiago avatar Feb 22 '21 13:02 dsantiago