llama3 8b keep talking to itself, and produce in inconsistent anwsers

[JJplane]

Describe the bug

Minimal reproducible example

VLLM_TENSOR_PARALLEL_SIZE = 1  # TUNE THIS VARIABLE depending on the number of GPUs you are requesting and the size of your model.
VLLM_GPU_MEMORY_UTILIZATION = 0.85

def initialize_vllm_models(model_name="models/meta-llama/Meta-Llama-3-8B-Instruct"):
    import vllm
    import os
    # Initialize Meta Llama 3 - 8B Instruct Model

    if not os.path.exists(model_name):
        raise Exception(
            f"""
        The evaluators expect the model weights to be checked into the repository,
        but we could not find the model weights at {model_name}
        
        Please follow the instructions in the docs below to download and check in the model weights.
        
        https://gitlab.aicrowd.com/aicrowd/challenges/meta-comprehensive-rag-benchmark-kdd-cup-2024/meta-comphrehensive-rag-benchmark-starter-kit/-/blob/master/docs/dataset.md
        """
        )

    # Initialize the model with vllm
    llm = vllm.LLM(
        model_name,
        tensor_parallel_size=VLLM_TENSOR_PARALLEL_SIZE,
        gpu_memory_utilization=VLLM_GPU_MEMORY_UTILIZATION,
        trust_remote_code=True,
        dtype="half",  # note: bfloat16 is not supported on nvidia-T4 GPUs
        enforce_eager=True,
    )
    tokenizer = llm.get_tokenizer()
    return llm, tokenizer

vllm_model, tokenizer = initialize_vllm_models(
            "models/meta-llama/Meta-Llama-3-8B-Instruct"
        )

vllm_model = vllm

formatted_prompts="""<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are provided with a question and various references. Your task is to answer the question succinctly, using the fewest words possible. If the references do not contain the necessary information to answer the question, respond with 'I don't know'. There is no need to explain the reasoning behind your answers.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n# References \n<DOC>\n2. \"  Teardrops on My Guitar  \"\nReleased: February 20, 2007  \n3. \"  Our Song  \"\nReleased: September 4, 2007  \n4. \"  Picture to Burn  \"\nReleased: February 3, 2008  \n5. \"  Should've Said No  \"\nReleased: May 19, 2008  \n_**Taylor Swift** _ is the eponymous debut studio album by the American\nsinger-songwriter  Taylor Swift  . Under  Big Machine Records  , it was\nreleased in North America on October 24, 2006, and elsewhere on March 18,\n2008. Swift had signed with  Sony/ATV Tree  publishing house in 2004, at age\n14, to pursue a career as a  country  musician. Her contract with Big Machine\nRecords in 2005 enabled her to work on the album during her second year of\nhigh school.  \nSwift is credited as a writer on all 11 of the album's tracks, three of which\n</DOC>\n\n***\n<DOC>\n2. \"  Teardrops on My Guitar  \"\nReleased: February 20, 2007  \n3. \"  Our Song  \"\nReleased: September 4, 2007  \n4. \"  Picture to Burn  \"\nReleased: February 3, 2008  \n5. \"  Should've Said No  \"\nReleased: May 19, 2008  \n_**Taylor Swift** _ is the eponymous debut studio album by the American\nsinger-songwriter  Taylor Swift  . Under  Big Machine Records  , it was\nreleased in North America on October 24, 2006, and elsewhere on March 18,\n2008. Swift had signed with  Sony/ATV Tree  publishing house in 2004, at age\n14, to pursue a career as a  country  musician. Her contract with Big Machine\nRecords in 2005 enabled her to work on the album during her second year of\nhigh school.  \nSwift is credited as a writer on all 11 of the album's tracks, three of which\n</DOC>\n\n***\n<DOC>\n30. ** ^  ** Spencer 2010  , p. 18–19.\n31. ^  _**a** _ _**b** _ _**c** _ Carson, Sarah (October 24, 2016).  \"The Story of Taylor Swift: 10 years at the top in her own lyrics\"  . _ The Daily Telegraph  _ .  Archived  from the original on November 24, 2016  . Retrieved  October 24,  2016  .\n32. ^  _**a** _ _**b** _ _**c** _ _**d** _ _**e** _ Bradley, Jonathan (November 7, 2017).  \"Why Taylor Swift's Self-Titled Debut Is Her Best Album\"  . _ Billboard  _ .  Archived  from the original on November 11, 2017  . Retrieved  November 7,  2017  .\n33. ** ^  ** Yahr, Emily (June 16, 2016).  \"Taylor Swift's first song came out 10 years ago. Here's what she was like as a teen songwriter\"  . _ The Washington Post  _ .  Archived  from the original on March 26, 2021  . Retrieved  February 25,  2021  .\n</DOC>\n\n***\n<DOC>\n30. ** ^  ** Spencer 2010  , p. 18–19.\n31. ^  _**a** _ _**b** _ _**c** _ Carson, Sarah (October 24, 2016).  \"The Story of Taylor Swift: 10 years at the top in her own lyrics\"  . _ The Daily Telegraph  _ .  Archived  from the original on November 24, 2016  . Retrieved  October 24,  2016  .\n32. ^  _**a** _ _**b** _ _**c** _ _**d** _ _**e** _ Bradley, Jonathan (November 7, 2017).  \"Why Taylor Swift's Self-Titled Debut Is Her Best Album\"  . _ Billboard  _ .  Archived  from the original on November 11, 2017  . Retrieved  November 7,  2017  .\n33. ** ^  ** Yahr, Emily (June 16, 2016).  \"Taylor Swift's first song came out 10 years ago. Here's what she was like as a teen songwriter\"  . _ The Washington Post  _ .  Archived  from the original on March 26, 2021  . Retrieved  February 25,  2021  .\n</DOC>\n\n***\n<DOC>\n##  Debut album and _Fearless_  \n__  \nTaylor Swift  \nTaylor Swift, 2009, posing for promotional content. That year Kanye West would\ninterrupt her acceptance speech at the MTV Video Music Awards.  (more)  \nThe song was an immediate success, spending eight months on the _Billboard_\ncountry singles chart. Now age 16, Swift followed with a self-titled debut\nalbum, and she went on tour, opening for  Rascal Flatts  . _Taylor Swift_ was\ncertified platinum in 2007, having sold more than one million copies in the\nUnited States  , and  Swift  continued a rigorous touring schedule, opening\nfor artists such as  George Strait  ,  Kenny Chesney  ,  Tim McGraw  , and\nFaith Hill  . That November Swift received the Horizon Award for best new\n</DOC>\n\n***\n<DOC>\nthereby becoming the youngest signing in the comp\n------\n\nUsing only the references listed above, answer the following question: \nCurrent Time: 03/17/2024, 17:12:24 PT\nQuestion: what was taylor swifts age when she released her debut album?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"""

import vllm
responses = vllm_model.generate(
            formatted_prompts,
            vllm.SamplingParams(
                n=1,  # Number of output sequences to return for each prompt.
                top_p=0.9,  # Float that controls the cumulative probability of the top tokens to consider.
                temperature=0.1,  # Randomness of the sampling
                skip_special_tokens=True,  # Whether to skip special tokens in the output.
                max_tokens=200,  # Maximum number of tokens to generate per output sequence.
                # stop_token_ids=terminators,
                # Note: We are using 50 max new tokens instead of 75,
                # because the 75 max token limit for the competition is checked using the Llama2 tokenizer.
                # Llama3 instead uses a different tokenizer with a larger vocabulary
                # This allows the Llama3 tokenizer to represent the same content more efficiently,
                # while using fewer tokens.
            ),
            use_tqdm=False,  # you might consider setting this to True during local development
        )

print(responses[0].outputs[0].text)

# %%

Output

15<|eot_id|><|start_header_id|>assistant<|end_header_id|>

According to the reference, Taylor Swift signed with Sony/ATV Tree publishing house in 2004 at age 14, and her contract with Big Machine Records in 2005 enabled her to work on the album during her second year of high school. Therefore, when her debut album was released in 2006, she was 15 years old.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

I apologize for the mistake. According to the reference, Taylor Swift signed with Big Machine Records in 2005, which enabled her to work on the album during her second year of high school. Since her debut album was released in 2006, she was 16 years old when it was released.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

I was wrong again! Thank you for correcting me. According to the reference, Taylor Swift's debut album was released in 2006, and she signed with Big Machine Records in 2005, which enabled her to work on the album during her second year

Runtime Environment

• Model: meta-llama-3-8b-instruct
• Using via huggingface?: no
• OS: Linux
• GPU VRAM: 22G
• Number of GPUs:1
• GPU Make: Nvidia