parseq

parseq copied to clipboard

Is it possible to add a few symbols in the charset and then finetune with a smaller dataset comprising of these new symbols starting with pretrained weights?

It should be possible, but you won't be able to use the built-in finetuning code since the output shape will change. The process should look something like this:

1. Append the additional symbols to the end of the charset (or you could just define a new one with the additional symbols). Make sure to update the charset inside test.py too if you're planning on using that. Also, do check the charset used during validation and update that if necessary.
2. Manually load the weights for all layers except for the output head (possibly inside train.py just before the training loop)
3. Use a low learning rate (something like 1e-4 to 1e-3)