How to run evaluation on the validation set?

Hi,

Is is possible to provide a set of instructions to run evaluation on the validation set?

From the README:

test_eval.json is used for evaluation. test_llama.json is used for training

However, when I run:

python demo.py --llama_dir /path/to/llama_model_weights --checkpoint /path/to/pre-trained/checkpoint.pth --data ../test_llama.json  --output ../output.json --batch_size 4 --num_processes 8

python evaluation.py --root_path1 ./output.json --root_path2 ./test_eval.json

I face this random UUID issue. I follow the exact instructions, so I'm not sure why this doesn't work.

For further diagnosis, following the FAQ which say I should run inference on the validation set, I ran:

python convert2llama.py

and changed this line to v1_1_val_nus_q_only.json and output to val_llama.json

And then did:

python demo.py --llama_dir /path/to/llama_model_weights --checkpoint /path/to/pre-trained/checkpoint.pth --data ../val_llama.json  --output ../output_val.json --batch_size 4 --num_processes 8

python evaluation.py --root_path1 ./output_val.json --root_path2 ./v1_1_val_nus_q_only.json

But this doesn't work either, and shows the same UUID error.

Could you post the UUID error here? Are you running eval on your local env or our test server?