different seed not changing outputs

[codecodebear]

Hi,

I set different seed values in config, but this doesn't change results' metric scores. I also looked at the intermediate data, retrieved passages, pred answers, etc remain the same for different seeds. I used vllm as engine.

Not sure where I missed. Can you help me? Thank you!

[ignorejjj]

The retrieval result in this process should be fixed and not affected by randomness. Have you ever tried setting a larger temperature in generator config? This should result in different outputs.

And it should be noted that in vllm the seed should be set separately in the generator config (see https://docs.vllm.ai/en/latest/dev/sampling_params.html). If the temperature setting above is invalid, you can try adding seeds in the generator config.