Repeated text for longer prompts.

[PawelFaron]

Below is my simple code for text generation. The problem is that when the prompt is little bugger (as is that case) the generation goes wild and just keeps repeating itself after several tokens. When I have smaller prompts and it produces smaller output it works great. It there something I'm missing? I've tried setting repetition penalty, different temperatures, top-k.

from ctransformers import AutoModelForCausalLM

model_id = 'TheBloke/Llama-2-7B-Chat-GGML'
# model_file = 'llama-2-7b-chat.ggmlv3.q8_0.bin'
model_file = "llama-2-7b-chat.ggmlv3.q6_K.bin"
model = AutoModelForCausalLM.from_pretrained(model_id, model_file=model_file)

def build_llama2_prompt(messages):
    startPrompt = "<s>[INST] "
    endPrompt = " [/INST]"
    conversation = []
    for index, message in enumerate(messages):
        if message["role"] == "system" and index == 0:
            conversation.append(f"<<SYS>>\n{message['content']}\n<</SYS>>\n\n")
        elif message["role"] == "user":
            conversation.append(message["content"].strip())
        else:
            conversation.append(f" [/INST] {message['content'].strip()}</s><s>[INST] ")

    return startPrompt + "".join(conversation) + endPrompt


system = """
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.

If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
"""

messages = [
    {"role": "system", "content": system}
]

prompt = """
This is a initial list of best movies from 90'. Add 5 more.

1. The Shawshank Redemption (1994) - a drama film about two prisoners who become friends and find hope and redemption.
2. The Lion King (1994) - an animated movie about a young lion who must assume his rightful place as king of the Pride Lands.
3. Jurassic Park (1993) - a science fiction film set on an island where dinosaurs are brought back to life through genetic engineering.
4. Schindler's List (1993) - a historical drama based on the true story of Oskar Schindler, a German businessman who saves the lives of thousands of Jews during the Holocaust.
5. Pulp Fiction (1994) - a crime film that interweaves several stories about mobsters, hitmen, and other criminals in Los Angeles.
6. Forrest Gump (1994) - a drama film about a man with a low IQ who finds himself involved in some of the most significant events of the 20th century.
7. The Silence of the Lambs (1991) - a crime thriller about an FBI agent who seeks the help of a serial killer to solve a series of murders.
8. Terminator 2: Judgment Day (1991) - a science fiction film about a cyborg assassin sent back in time to protect a young John Connor future
"""
messages.append({"role": "user", "content": prompt})
prompt = build_llama2_prompt(messages)

for word in model(prompt, stream=True, temperature=0.7, max_new_tokens=2048):
    print(word, end='')