api not producing the same result as ui - WizardLM

[jonadshead]

Describe the bug

I cannot get an expected result when trying to run this model via the api - https://huggingface.co/ausboss/WizardLM-13B-Uncensored-4bit-128g

When asking "hey, whats 1+1?" the ui responds with "2" as you'd expect. But when trying the same prompt via the api the response is " \n släktet".

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

llama type, 4bit, 128g. In UI, mode is instruct and instruction template is WizardLM.

import requests

def run(prompt: str) -> str:
    request = {
        'prompt': prompt,
        'max_new_tokens': 200,
        'do_sample': True,
        'temperature': 0.7,
        'top_p': 0.5,
        'typical_p': 1,
        'repetition_penalty': 1.2,
        'top_k': 40,
        'min_length': 0,
        'no_repeat_ngram_size': 0,
        'num_beams': 1,
        'penalty_alpha': 0,
        'length_penalty': 1,
        'early_stopping': False,
        'seed': -1,
        'add_bos_token': True,
        'truncation_length': 2048,
        'ban_eos_token': False,
        'skip_special_tokens': True,
        'stopping_strings': []
    }

    response = requests.post(f'http://127.0.0.1:5000/api/v1/generate', json=request)

    if response.status_code == 200:
        return response.json()['results'][0]['text']

    return ""

prompt = "hey, whats 1+1?"
response = run(prompt)
print(response) # the expected result via ui is '2'. But the result via api is ' \n släktet'.

Screenshot

No response

Logs

N/A

System Info

N/A

[MikoAL]

Have you tried setting the seed to the same thing? The seed you set in the "Reproduction" section is marked as -1, which means random seed. That could be why? idk, haven't tested it myself

[jonadshead]

Have you tried setting the seed to the same thing? The seed you set in the "Reproduction" section is marked as -1, which means random seed. That could be why? idk, haven't tested it myself

that is irrelevant. If you ask a question like whats 1+1 it should contain 2 every time. which it seems to do in the ui. I have tried multiple times on both ui and api. but the api keeps responding with only släktet which mean "the genus" in Swedish. So it's clearly nothing to do with seed, its a nonsense response :)

[Kwisss]

but the api keeps responding with only släktet which mean "the genus" in Swedish. So it's clearly nothing to do with seed

Are u swedish? do you by any chance have some swedish contenxet? haha

but on the serious note, while this bug stopping you, (I'm not a dev) You could try TheBloke version of WizardLM-13B-Uncensored-4bit-128g on huggingface.

[github-actions[bot]]

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.