performer-pytorch icon indicating copy to clipboard operation
performer-pytorch copied to clipboard

context-specific embeddings from language model?

Open rainwala opened this issue 4 years ago • 0 comments

Hi, thank you very much for all of your amazing work to implement bleeding-edge attention models in pytorch. This is a question, not an issue. Does the PerformerLM language model learn context-specific embeddings for the tokens? I would like to use this model for a fine-tuning task -- training the language model on a huge dataset, then using part of the trained language model plus some more layers to fine-tune on a downstream task. I think this might be powerful, but only if the PerformerLM trains context-specific embeddings. I know this kind of thing is possible with BERT-like architectures, but I'm interested in much longer sequences, which is why I would like to try it with PerformerLM.

rainwala avatar Feb 23 '21 17:02 rainwala