[Generation, Question] Why does the `seed` have to be the same in different processors (`Llama.build`)?

[keli-wen]

✨ First of all, the Llama project is a tremendous asset to the entire GenAI field. I deeply appreciate the contributions of both the Llama team and the open-source community.

Description

I've recently been delving into the source code of the Llama project and I happen to have a question. Specifically, why seed must be the same in all processes.

This line of code is located in the Llama.build function at: https://github.com/meta-llama/llama/blob/main/llama/generation.py#L94-L95.

Regarding this code, there are only two versions of changes:

• The original version: https://github.com/meta-llama/llama/blob/6d4c0c290aeec1fa4399694fefb864be5a153bb6/llama/generation.py#L71-L72
• The current version: updated to the seedable version in this PR https://github.com/meta-llama/llama/pull/779.

However, I have a question: why must the seed be equal? What would happen if they are not equal? I tried searching for "why must random seeds be equal in model parallelism?", but I couldn't find any relevant information.

From an engineering perspective, I think it might be related to reproducibility? But I want to know if having different random seeds for each processor would result in a catastrophic outcome? (Because the current comment makes it seem like a very serious issue)

The current docstring lacks an explanation for seed and fails in the ruff linter. (I believe the lack of comments is also one of the reasons contributing to the confusion.)

I really hope to receive help from Meta or anyone else. I would also be happy to provide the correct docstring for seed after resolving this issue to eliminate any potential confusion.