How to extract pre-final layer embeddings ?

[xinformatics]

Hi, Could you please tell if its possible to extract embeddings from the pre-final layers similar to Facebook's ESM1v / ESM1b models?

Thank you