Stream API example seems broken.

[xchiptune]

Running the API example for the stream mode I get the following error:

FileNotFoundError: [Errno 2] No such file or directory: 'softprompts/What I would like to say is the following: .zip'

The non-stream mode one works fine. Think maybe a recent update changed the form format? There's no mention of it even being possible to upload a softprompt in the API example.

I'm running the webui with python server.py --load-in-4bit --model llama-13b --listen

[oobabooga]

I can confirm the issue.

[thot-experiment]

Same bug, anyone know which commit this broke on? Was working maybe 2 or 3 days ago. I'll look into this. I believe going forward we should probably switch to a versioned API (so that we don't break backwards compat) and avoid using an array for the input (unless there's some reason to do so, it seems very confusing not to use a dict so that the labels are obvious)

I'm unfamiliar with the code here but I'll dig around a little bit to see if I can figure out what the issue is here as well as see what it would take to add a /v1/* endpoint that accepts a dictionary, as well as providing default values for all unsubmitted fields to allow for simpler use i.e. submitting only

{
    "prompt": "Factual Q/A\nQuestion: What is the atomic number of lithium?\nAnswer: "
}

and having the rest of the configuration filled in by whatever default is loaded in at the time.

Sorry if any of this is already implemented of if I'm missing something obvious.

[thot-experiment]

So it looks like this API is something that's autogenerated by Gradio itself (sorry again for the naivete here I really have no idea what's going on) and because of the recent changes in the UI the shape of this API has changed as well. (I think we should try to figure out how to avoid this in the future)

For the time being here's an example of a working payload so anyone blocked by this can hack together working code for the time being and perhaps to give people an idea where to look next time this breaks.

Everything behaves the same as the example with the exception of the two lines with comments.

{
  "fn_index": 9, //this was '7' originally and now must be '9'
  "data": [
    "Common sense questions and answers\\n\\nQuestion: What is the atomic number of lithium?\\nFactual answer:",
    200,
    true,
    1.99,
    0.18,
    1,
    1.15,
    1, //this parameter is new and represents the "encoder_repetition_penalty"
    30,
    0,
    0,
    1,
    0,
    1,
    false
  ],
  "session_hash": "f420f69f"
}

P.S. Really seems not good to add a parameter in the middle of an array of nameless parameters, i sorta assume this API is meant purely as an M2M interface internally for Gradio?

[LTSarc]

@thot-experiment the parameters are named in api-example-stream.py

I am going to assume that simply hacking the FN index to be 9 will fix it, given that otherwise the numbers match up with that working payload.

[oobabooga]

@thot-experiment I really really would like to use **kwargs in the main functions, which would allow the API to pass parameters as a dictionary as you mentioned. The problem is that gradio does not seem to accept that.

Specifically, the offending function is

def generate_reply(question, max_new_tokens, do_sample, temperature, top_p, typical_p, repetition_penalty, encoder_repetition_penalty, top_k, min_length, no_repeat_ngram_size, num_beams, penalty_alpha, length_penalty, early_stopping, eos_token=None, stopping_string=None):

in modules/text_generation.py.

This long function definition causes modules/chat.py to be very messy with several other long function definitions.

[oobabooga]

Also, if you are interested in APIs, make sure to also check https://github.com/oobabooga/text-generation-webui/pull/342

It is possible to create custom APIs using extensions.

[cpwilson1989]

This appears to have raised its head again, but with the corrent fn_index now being 11!

[Anonym0us33]

\modules\chat.py", line 437, in load_character
    file_contents = open(filepath, 'r', encoding='utf-8').read()
FileNotFoundError: [Errno 2] No such file or directory: 'characters\\instruction-following\\TavernAI-Fisherwoman Minori.json'

not sure if this bug is continuing or if this is new but i don't want to waste anyone's* time with a new issue above happens a lot of the time when loading cards.

>dir /d
Example.png                      
Example.yaml                       TavernAI-Emma.json                
[instruction-following]            TavernAI-Emma.png

\instruction-following>dir /d
 Alpaca.yaml           Vicuna.yaml
[..]                  Open Assistant.yaml

[github-actions[bot]]

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.