nanoGPT icon indicating copy to clipboard operation
nanoGPT copied to clipboard

Published 20 hours ago • verified publisher icon karpathy

Implement ROPE positional encodings

Open devinbot opened this issue 11 months ago • 1 comments

This PR includes the implementation of ROPE positional encodings and adjustments to the training script for the Shakespeare dataset.

devinbot avatar Mar 08 '24 04:03 devinbot

Nice work. Curious how much loss did you get on this?

xianbaoqian avatar May 02 '24 04:05 xianbaoqian