Training Time of Reranker

[Impavidity]

trafficstars

Thanks for the great work and the open-source models. BTW, I am quite interested in the following questions.

1. Total time to train the LLMReranker, such as Gemma and MiniCPM, under what kind of hardware.
2. Max length for the query/passage and batch size when training with LLM Reranker.

Many thanks!

[545999961]

We trained for 4 days on 8 * 40G A100 GPUs. During training, the total length of query plus passage was 1024, and the batch size was 128.

[Impavidity]

Thank you for you quick follow up. Sorry I have another question: How many epoch you trained on all m3+fever+quora data? Do you do any downsampling?

[545999961]

Training for 1-2 epochs is enough.

[Impavidity]

Thanks for the reply. Sorry I have more questions here.

1. for long context example (e.g. length > 1k), do we decrease the batch size during the training? If it is, is this done automatically?
2. During the training, is left padding used or right padding (for padding_side of tokenizer)

[545999961]

1. It will truncate long contexts, so there is no need to decrease the batch size.
2. Follow the tokenizer's raw padding side.