Facing error if its more than 512 tokens

[yuvarajvc]

If i want to use more than 512 tokens bert is not allowing and i am facing an error, can anyone help me how to use bigger database which is having 2000+ tokens.

[brunnurs]

That is indeed a problem. You can use a more recent transformer (e.g. BART) which supports up to 1024 tokens, but you could also try to simplify your schema - most probably there are quite a few tables / columns which are purely technical and a user will never use.

[yuvarajvc]

Thanks for your reply, we are using larger database so the token are going beyond 512, we are planning to do exact column and table match as well as partial column match and table match and will pass only those tables and columns to BERT MODEL, do you think this approach will work or we have to send all the tables and columns