How to train models with contrastive losses, masks, etc.

[18liumin]

[n1o]

I would like to see the actual pretraining of CodeT5, I am reading up on the Identifier Tagging objective, where you transform the contextual representation of the encoder into a vector of probabilities. Unfortunately there no discussion on how, I assume there is an ProjectionLayer with an L2 norm that gets fed into a sigmoid. I came to look up the code, but I either to struggle to find it, or I am just blind.