mamba
mamba copied to clipboard
Add inputs_embeds as alternative for input_ids
Adding to the model.forward inputs_embeds (same name as HF models use) to give the model token embedding directly rather than token ids
The main use case is training soft prompts without adding N tokens to tokenizer that never get outputted(and then jumping through hoops to train them only without touching existing tokens)
(ETA: from more recent example: techniques like neft are much easier to do when it's possible to pass embedding directly)