Florence-2: How to add custom tokens during fine-tuning training?

[lixumin-zai]

This is the official way to add:

# processing_florence2.py
tokens_to_add = {'additional_special_tokens': \
    tokenizer.additional_special_tokens + \
    ['<od>', '</od>', '<ocr>', '</ocr>'] + \
    [f'<loc_{x}>' for x in range(1000)] + \
    ['<cap>', '</cap>', '<ncap>', '</ncap>','<dcap>', '</dcap>', '<grounding>', '</grounding>', '<seg>', '</seg>', '<sep>', '<region_cap>', '</region_cap>', '<region_to_desciption>', '</region_to_desciption>', '<proposal>', '</proposal>', '<poly>', '</poly>', '<and>']}

The training was added in this way, and the model was adjusted

model.resize_token_embeddings(len(processor.tokenizer))

The model's output is very good at the beginning, but the latter part looks like gibberish