Convert-PolyAI-Torch

Convert-PolyAI-Torch copied to clipboard

Cosine annealing vs cosine similarity clarification

1

really not clear from paper: 'computed as cosine similarity with annealing between the encodings hx and hy. It starts at 1 and ends atp d, linearly increasing over the first...

jordiclive

jordiclive

jordiclive

probs want no biases, stop model bloating

jordiclive

need to check ive implemented label smoothing with how authors how they label smoothed their objective sampling as objective fn includes negative sampling.

jordiclive

currently using torch gelu. fast gelu in paper

jordiclive

implement BPE from scratch with unk tokens hashed (although may achieve worse results on downstream tasks) as # perhaps not as general as bpemb's 25000.model

jordiclive