Error in logits_processors.py - OutputNumbersTokens().__call__() for some models

[bs7280]

Hello: When running the tiiuae/falcon-7b model, I get no issue using the package as intented. But some models, such as tiiuae/falcon-rw-1b will get an error in OutputNumbersTokens().call() like below:

The expanded size of the tensor (50304) must match the existing size (50257) at non-singleton dimension 1. Target sizes: [1, 50304]. Tensor sizes: [50257]

I've been trying to debug this on my own but have not figured out why sometimes self.allowed_mask and scores sometimes have mismatching shapes (depending on model) that will cause the above error when trying to run:

self.allowed_mask.expand_as(scores)

[noamgat]

I thought this was a bug in transformers or in the specific model, but I tried modifying the example notebook in lm-format-enforcer ( https://github.com/noamgat/lm-format-enforcer/blob/main/samples/colab_llama2_enforcer.ipynb ) which achieves a similar goal, to use tiiuae/falcon-rw-1b and it worked.

So I'm not sure

[Vincentyua]

Hi, I encountered the same problem where there was a size mismatch when running other models. Do you know how to solve it?