models icon indicating copy to clipboard operation
models copied to clipboard

[FEA] Design doc for the masking block

Open sararb opened this issue 3 years ago • 0 comments

Motivation

Training with masked input sequences has proven very effective for session-based and sequential tasks (Bert4Rec, SaSRec, and Transformers4Rec). As a first step towards integrating sequential models, Merlin Models should support masking classes.

Your contribution

Create a design doc that includes the following point:

  • Advantage of using masking training tasks.
  • Define four common tasks: Causal Language Modeling (CLM), Masked LM, Permutation LM, and Replacement token detection.
  • List the different scenarios to use masking.
  • Propose solutions to integrate the masking in the input block of Merlin Models.

Link to the google doc: https://docs.google.com/document/d/13rACJLAjjw7fWems6gJG9i3cg0ViG8JUZMBGcZz-awI/edit#

sararb avatar Jun 13 '22 13:06 sararb