Text generation yields different outputs despite temperature = 0.0

[ScTof]

Describe the bug Even though temperature is set to 0.0 in the configs, GPT-NeoX' generate.py (interactive mode) produces different results different outputs for the same input submitted multiple times in a row.

To Reproduce Steps to reproduce the behavior:

1. Use the attached config files
2. Run python deepy.py generate.py PATH/TO/CONFIGS_DIR/*
3. Submit a query (e.g., "This is a test:")
4. Resubmit the same query
5. Observe the different outputs

Expected behavior As far as I'm concerned, setting temperature to 0.0 should make GPT-NeoX produce the same output for the same input, no matter how often it is submitted. At the moment, it yields the same output only if the model is reloaded after each query. Setting reload = false or reload = true doesn't make a difference.

Environment (please complete the following information):

• GPUs: 2
• Configs: See attachments (notably, set temperature = 0.0 and recompute = true) 20B.yml.txt local_setup_20B.yml.txt text_generation.yml.txt