finetune-transformer-lm icon indicating copy to clipboard operation
finetune-transformer-lm copied to clipboard

Published 20 hours ago • verified publisher icon openai

Question about the shape of `X_train`

Open FrankWork opened this issue 6 years ago • 1 comments

X_train = tf.placeholder(tf.int32, [n_batch_train, 2, n_ctx, 2]) xmb[:, :, :, 1] = np.arange(n_vocab+n_special, n_vocab+n_special+n_ctx) why there is a channel of additional tokens?

FrankWork avatar Jul 02 '18 10:07 FrankWork

Problem solved! This part of the xmb is used for the learned positional encoding. https://github.com/huggingface/pytorch-openai-transformer-lm/issues/12#issuecomment-401770634

FrankWork avatar Jul 02 '18 11:07 FrankWork