AnswerDotAI
Doesn't support input vector? There is no inputs_embeds
in the parameter list of forward, there is no inputs_embeds parameter, so it doesn't support inputing vector (embedding vector not token id)?
To be compatible with Bert-base/large, need a new parameter inputs_embeds to accept vectors as input.
Hello,
Indeed, BertModel allows to pass inputs_embeds instead of input_ids, which effectively skip the embedding lookup layer and directly use the input tensor.
This could be added to ModernBERT by adding the same logic to ModernBertModel and forwarding it to ModernBertEmbeddings, which would end up being only a norm + dropout layer in such cases. Every subclass would also need to have this parameter and forward it to their respective ModernBertModel.
May I ask what is your use case for this? For VLM (or LLM, for that matter), I get that you might want to have some liberty on what you can feed the model, but here I am not sure I see why you would not use the embeddings layer directly? It should be added anyways for retro-compatiblity, but I am curious about todays use cases.
I have done a research for binary file identification, originally it is based on bert-base/large. I would like to transfer it to ModernBert to determine the potential of our methods.
Cool, thanks for the input! FYI, this feature is already being implemented to support Contextual Document Embeddings model in this PR (I did not realize this was how it was implemented in transformers), so it should be available soon!
Hello, closing this as the PR has been merged. Feel free to re-open if something is not working!