models
models copied to clipboard
[FEA] Design doc for the masking block
Motivation
Training with masked input sequences has proven very effective for session-based and sequential tasks (Bert4Rec, SaSRec, and Transformers4Rec). As a first step towards integrating sequential models, Merlin Models should support masking classes.
Your contribution
Create a design doc that includes the following point:
- Advantage of using masking training tasks.
- Define four common tasks: Causal Language Modeling (CLM), Masked LM, Permutation LM, and Replacement token detection.
- List the different scenarios to use masking.
- Propose solutions to integrate the masking in the input block of Merlin Models.
Link to the google doc: https://docs.google.com/document/d/13rACJLAjjw7fWems6gJG9i3cg0ViG8JUZMBGcZz-awI/edit#