WANG Yue

Hi, for each stage (either MSP or NTP task) of pretraining, we employ a small proprotion of training data as the held-out validation set and monitor the corresponding loss (either...

Hi, I remember that the training portion is not the same as non-valid/test portion and it is actually smaller. You can try to verify this. For data filtering, we might...

Hi, for embedding exaction with CodeT5, we suggest to follow the BART manner by feeding the sequences to both encoder and decoder. Then you can employ either the last decoder...

You can send me an email to request the twitter dataset and promise do not disclose it publicly (considering the copyright issues). Thanks.

Hi, although CodeT5 is pretrained with single functions, it should be able to transfer to encode multiple methods, which is however not tested yet.

Hi there, we have provide an example [finetuning script](https://github.com/salesforce/CodeT5/blob/main/CodeT5%2B/tune_codet5p_seq2seq.py) and please see [here](https://github.com/salesforce/CodeT5/blob/main/CodeT5+/README.md#how-to-finetune-using-your-own-data) for more details. For bigger models such as 2B and 6B, please use Deepspeed for training acceleration.

你好，你的邮件我已收到，我会尽快查看~~~

Hi there, `` is also a special token and here this is a kind of unexpected output. We would suggest to use `codet5p-220m` and `codet5p-770m` models in the finetuning setting....

Hello, may I know which model do you use? For CodeT5+ 2B/6B/16B, they are further finetuned on Python code and more suitable for Python code generation/completion.

HI TejaswiniiB, we did not test it but believe it should give proper embeddings.