Rangehow

我发现了原因，是因为只有这个instruct模型的tokenizer还把eos设置为预训练的eos token，其他系列的模型都正常设置了。 这可能还有一个问题，做ZERO类型RL的探索很多都是从base model开始的，所以对于base模型来说，需要额外将他们的eos token设置为与开源instruct模型的后训练eos token对齐的设置吗，还是就直接采用预训练的eos token设置就行？

一样的问题

> > ### 🐛 Describe the bug > > When using different generation configurations, such as top_k=1 or temperature=0 (while keeping other settings unchanged), why do the generated results change?...

> `benchmarks/benchmark_throughput.py` does not support data parallel right now. offline inference with data parallel is quite different, you cannot just call one `LLM.generate`, but multiple instances need to call `LLM.generate`...

any update on this？ met this on transformers 4.46.2. I use self specified device_map, this same code with qwen is okay ```python from niuload import balanced_load import torch from transformers...

> Could you share a reproducer of the issue that you are facing @rangehow ? I didn't understand what you meant with the qwen model. With which model do you...

> Could you try casting your input arrays as int8? > > In randint, just add `dtype=np.int8` Thank you very much for your prompt reply. However, my use case requires...

Thank you very much for your help. I am using the repository you developed to process the dataset after tokenization of a large language model. More specifically, it is https://arxiv.org/pdf/2401.17377...

Thank you for your time. I will test the performance difference between this interface and my simple Python implementation :)

> We can try, but I'm not 100% sure we can, since generally we assume that anything related in there will be cast via `amp` for loss calculation properly.我们可以尝试，但我不是100%确定我们可以，因为通常我们假设那里的任何相关内容都将通过 `amp`...