REQUEST guide on using the API

[GamingDaveUk]

Description

Pretty familar with c# coding, java, Kotlin... all the same breed really.... but nover really used an api that did not come with prepakaged code lol. All I want to know is how to do you send a prompt to a runnung copy of Oobabooga and recieve the generated text back, this would help for a c# project i am working on for fun and would help to get it used in ComfyUI (custom node, either by myself when i figure out how, or by someone WAY better who reads the api guide

Additional Context

Ideally how to add it to a c# project, or if i am asking something really simple that a hobby coder may just not know, where to look for similar things

[GamingDaveUk]

Not one for sitting ideal and waiting.... I (ok mostly chatgpt) came up with a function that does make a request of the oobabooga inside a c# application:

public static async Task<string> AskLocalAi(string prompt)
        {
            var httpClient = new HttpClient();

            var requestBody = new
            {
                prompt = prompt,
                length = 50,
                temperature = 0.5,
                stop = "\n"
            };

            var requestUri = "http://localhost:5000/api/v1/generate";

            var requestBodyJson = JsonConvert.SerializeObject(requestBody);

            var requestContent = new StringContent(requestBodyJson, Encoding.UTF8, "application/json");

            var response = await httpClient.PostAsync(requestUri, requestContent);

            var responseBody = await response.Content.ReadAsStringAsync();

            dynamic responseData = JsonConvert.DeserializeObject(responseBody);

            string generatedText = responseData.results[0].text;

            return generatedText;
        }

This works, though the responses are a little insane....I am on my work laptop so rather vram limited. Naturally the ip would need to be changed to the machine that is hosting the oobabooga... and I have no idea how to select a character for the api to use for the responses?

[LaaZa]

I think you just have to add the "Character" to your prompt manually. I mean that you manually manage the prompt since the prompt that you send is the only thing that the api uses.

If we take Vicuna as an example this is the template: <|user|> <|user-message|>\n<|bot|> <|bot-message|></s>\n

and you replace <|user|> with USER: and <|bot|> with ASSISTANT:

The prompt you send should always end with the ASSISTANT: <- note the space Then it will know to generate the answer as the assistant.

So for the first round it would look like this:

A chat between a user and an assistant.
USER: Hello!\nASSISTANT: 

Then you get generated text and you add it to the prompt for next round:

A chat between a user and an assistant.
USER: Hello!\nASSISTANT: Hello! How can I help you today?</s>\nUSER: I need help with an api.\nASSISTANT: 

At some point you need to remove older messages because of the context limit. If you just keep sending the prompt as is but it does not fit the context, the beginning of it will be cut off and you might lose the prompt at the beginning that guides the model to answer in a certain way. Same thing goes for actual character definitions that you might want to keep in the prompt at all times. Textgen doesn't do any of this for you in the api unlike in chat mode in the webui.

[github-actions[bot]]

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.

[hypersniper05]

One easy way is to activate the openai extension and then either create your own script to talk to the api or use one of the openai c# ones. Note this is not a endpoint to openai but rather your local llm using openai's api.