generate crashes with certain inputs

[asolano]

Greetings,

while testing GLM's generate.sh script we have found that certain inputs seem to cause a crash. Specifically the error is:

Traceback (most recent call last):
  File "/home/acb11899xv/GLM-130B/generate.py", line 221, in <module>
    main(args)
  File "/home/acb11899xv/GLM-130B/generate.py", line 214, in main
    generate_continually(process, args.input_source)
  File "/home/acb11899xv/miniconda3/envs/glm130b/lib/python3.9/site-packages/SwissArmyTransformer/generation/utils.py", line 83, in generate_continually
    func(raw_text)
  File "/home/acb11899xv/GLM-130B/generate.py", line 191, in process
    answers, answers_with_style, blanks = fill_blanks(raw_text, model, tokenizer, strategy)
  File "/home/acb11899xv/GLM-130B/generate.py", line 135, in fill_blanks
    unfinished = output.index(-1)
AttributeError: 'int' object has no attribute 'index'

It seems that in generate.py's fill_blanks code the output variable gets turned into an integer instead of a list

We are using the following (slightly changed parameters) generate.sh:

#!/bin/bash

eval "$(command conda 'shell.bash' 'hook' 2> /dev/null)"
conda activate glm130b

script_path=$(realpath $0)
script_dir=$(dirname $script_path)
main_dir=~/GLM-130B

source "${main_dir}/configs/model_glm_130b.sh"

SEED=42
MAX_OUTPUT_LENGTH=128 # 256
MIN_GEN_LENGTH=0
# BeamSearchStrategy args
NUM_BEAMS=1
LENGTH_PENALTY=1.0
NO_REPEAT_NGRAM=3
# BaseStrategy args
TEMP=0.01
TOPK=1
TOPP=0

ARGS="${main_dir}/generate.py \
       --seed $SEED \
       --mode inference \
       --sampling-strategy BaseStrategy \
       --out-seq-length $MAX_OUTPUT_LENGTH \
       --min-gen-length $MIN_GEN_LENGTH \
       --num-beams $NUM_BEAMS \
       --length-penalty $LENGTH_PENALTY \
       --no-repeat-ngram-size $NO_REPEAT_NGRAM \
       --temperature $TEMP \
       --top_k $TOPK \
       --top_p $TOPP \
       --output-path samples \
       $MODEL_ARGS \
       $*"

run_cmd="torchrun --nproc_per_node $MP_SIZE ${ARGS}"
eval ${run_cmd}

This input file works OK:

000	 Carol and her mom were picking carrots from their garden. Carol picked 29 and her mother picked 16. If only 38 of the carrots were good, how many bad carrots did they have?  Let's think step by step. [gMASK]
001	 Roger had 16 dollars. For his birthday he got 28 more dollars but spent 25 on a new game. How much money does he have now?  Let's think step by step. [gMASK]

But this other one does not:

000	Carol and her mom were picking carrots from their garden. Carol picked 29 and her mother picked 16. If only 38 of the carrots were good, how many bad carrots did they have? Let's think step by step. First, we need to know how many carrots were picked. We know that Carol picked 29 and her mother picked 16. So, we know that 29 + 16 = 45. Next, we need to know how many good carrots were picked. We know that 38 of the carrots were good. So, 38 = 45 – 16. Finally, we need to know how many bad carrots were picked. We know that 16 of the carrots were bad. So, 16 = 45 – 38. The answer is 17. Therefore, the answer is [gMASK]
001	Roger had 16 dollars. For his birthday he got 28 more dollars but spent 25 on a new game. How much money does he have now? Let's think step by step. First, he had 16 dollars. Then he got 28 more dollars. Now he has 44 dollars. But he spent 25 dollars on a new game. So he has 29 dollars left. Therefore, the answer is [gMASK]

It is clearly longer, but there seems to be code to detect a "text too long" and raise and exception in fill_blanks.

Any insight on the cause of this behavior would be much appreciated.

Best,

Alfredo