LASS icon indicating copy to clipboard operation
LASS copied to clipboard

Published 20 hours ago • verified publisher icon pkusjh

Question about mask language modeling

Open ZhaozwTD opened this issue 2 years ago • 1 comments

Hello, I have a problem with the following lines about the mlm/span mask: https://github.com/pkusjh/LASS/blob/43f4c5e76525120ead3097fc3952f719db6dc695/model/data_collator.py#L24 https://github.com/pkusjh/LASS/blob/43f4c5e76525120ead3097fc3952f719db6dc695/model/roberta_model.py#L95

Are these lines not executed in the code? Operations related to mask language modeling are not mentioned in the original paper.

Many thanks for your nice codes!

ZhaozwTD avatar Feb 09 '23 02:02 ZhaozwTD

Thank you for pointing this out! We do not use span mask and mlm loss in the current version so the lines you marked are not executed yet.

jhshen95 avatar Feb 09 '23 02:02 jhshen95