I want use `gpt-3.5-turbo` for nlp task

[joowon-dm-snu]

I sometimes use gpt-3.5-turbo for NLP tasks like text-davinci-003. Because it's cheaper and feels like it performs much better than Curie.

But there are some problems with this. In the current version of python-preview, if I use the chat backend, it forces the template to record the user's conversation. This increases the number of tokens I use, which is costly. Also, if I'm dealing with long texts, it hits token limit in no time. like example below.

This can be solved by putting a memorization setting in the PromptTemplateConfig class and modifying the semantic-kernel/python/semantic_kernel/orchestration/sk_function.py, as shown in the photo below. But I didn't open the PR because I'm not sure if it will match the direction Microsoft is looking at.

In the same vein, I'd like to see ChatCompletion imported via import_semantic_skill_from_directory the way it's done in the text-davinci-003. Currently I'm importing skills the following way and it feels unnatural, please let me know if I'm missing something.

def import_skills(
    kernel: sk.Kernel, skill_dir="./skills"
) -> Dict[str, sk.SKFunctionBase]:
    skills = {}

    for skill in os.listdir(skill_dir):
        if skill.endswith("Skill"):
            s = kernel.import_semantic_skill_from_directory(skill_dir, skill)
            skills[skill] = s

    skills["ChatSkills"] = {}
    skills["ChatSkills"][
        "ExtractInformationList"
    ] = extract_information.build_semantic_chat_function(kernel)

    return skills

I think using skprompt.yaml for prompt template instead of skprompt.txt would allow for a much freer use of the model.

[jjhenkel]

Hey @joowon-dm-snu! Thanks for the thoughtful feedback, let me summarize to make sure I'm capturing everything:

1. You'd like to be able to use chat backends without keeping conversation history (totally agree with that!)
2. You've tried adding a memorize flag, and that seems like a natural solution (also agree, I like the look of that, it feels natural)
3. You want to be able to easily import skills that use chat backends from a directory (this is easy for functions that use a text completion backend, but not for ones using chat backends yet)
4. You would like more flexibility in skprompt.txt (suggest a structured format like YAML)

---

I'd be happy to make sure 1, 2, and 3 all happen! I think those are very reasonable asks. I think (4) we'd have to think through more; specifically, what information we'd want to put with a text prompt if we were going to support a structured format (and whether that info should just be in the config.json file, instead)

Thanks again for all your time working with SK @joowon-dm-snu! The community feedback is invaluable for developing and refining this further!!

[dluc]

I might be wrong but I think the ask is about using the /chat endpoint the same way we use /completion, ignoring the chat history aspect, and having only one System message and one User message, using the Assistant message response. @joowon-dm-snu does that make sense?

e.g.

function = skprompt.txt

Summarize this: {{$input}}
and translate to French

function.InvokeAsync(input)

POST /v1/chat/completions
{
  "model": "gpt-3.5-turbo",
  "messages": [{"role": "user", "content": "Summarize this: [injected text]\nand translate to French"}]
}

result = function.InvokeAsync(input)

{
  ...
  "object": "chat.completion",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "...summary in French here...",
    },
    "finish_reason": "stop"
  }],
  "usage": { ...  }
}

result => ...summary in French here...

[joowon-dm-snu]

Sorry for the confusion, I should have written a little more structured, but it was 4am and I was a little tired :)

I think @jjhenkel summarized it well, it was all what I wanted to say. 💯 The @dluc example is also correct, in fact I'm writing code like @dluc 's example in my personal library.
I wasn't sure how I can explain that example well, so I only included the intent this time, but thanks for catching it. You got my mind. I originally wanted to say this as another feature request.

Thanks for great open-source. I like its philosophy and MS's code works. I'm preparing consultations for other companies with semantic-kernel and hope this open source grow more. So feel free to message me if you need a little help.

[tynguyen]

@dluc perhaps my question relates to what @joowon-dm-snu is raising. Is there any ongoing shot in enabling gpt-3.5-turbo in ITextCompletion so that we can CreateSemanticFunction with this model?

[dluc]

@tynguyen it's not planned but I agree it's very important. I'm currently thinking through a few options to see which one would be the most seamless integration. One thing to check is how the tokenization works with the chat endpoint, and whether there's a message size limit. I couldn't find any information online yet.

[tynguyen]

@dluc: I'm pleased to hear that. Regarding to the tokenization, is this what you're referring to? Source: https://platform.openai.com/docs/guides/chat/introduction

If this does not work, a trick in this is to set a conservative number (for example, 4000 instead of 4096) as the maximum total tokens limit (message + completion tokens) for gpt-turbo-3.5 and it's been working well. What do you think?

[lightningRalf]

@tynguyen it's not planned but I agree it's very important. I'm currently thinking through a few options to see which one would be the most seamless integration. [...]

If you think about adding this possibility, maybe you could extend the thought about the ability to extend local models as well? Thanks in advance.

[joowon-dm-snu]

@jjhenkel

I hope feature below implemented soon. This will help managing my skill directories more systemically.

3. You want to be able to easily import skills that use chat backends from a directory (this is easy for functions that use a text completion backend, but not for ones using chat backends yet)

If concept below is okay, I can add PR soon like #505

# this is original import_semantic_skill_from directory
    def import_semantic_skill_from_directory(
        self, parent_directory: str, skill_directory_name: str
    ) -> Dict[str, SKFunctionBase]:
        CONFIG_FILE = "config.json"
        PROMPT_FILE = "skprompt.txt"

        kernel = self.kernel()
        ...
            config = PromptTemplateConfig()
            config_path = os.path.join(directory, CONFIG_FILE)
            with open(config_path, "r") as config_file:
                config = config.from_json(config_file.read())

            # Load Prompt Template
            with open(prompt_path, "r") as prompt_file:
                template = PromptTemplate(
                    prompt_file.read(), kernel.prompt_template_engine, config
                )
       ...

# this is my proposal for semantic chat skills
    def import_semantic_chat_skill_from_directory(
        self, parent_directory: str, skill_directory_name: str
    ) -> Dict[str, SKFunctionBase]:
        CONFIG_FILE = "config.json"
        USER_QUERY_FILE = "sk_user_query.txt"
        PROMPT_FILE = "sk_chat_prompt.txt"
        SYSTEM_MSG_FILE = "sk_system_prompt.txt"

        kernel = self.kernel()
        ...
            with open(user_query_path, "r") as prompt_file:
                user_query_msg = user_query_path.read()

            template = ChatPromptTemplate(
                user_query_msg, # if no user_query_file, use "{{$input}}"
                kernel.prompt_template_engine,
                config,
            )

            # Load System Message
            with open(system_msg_path, "r") as prompt_file:
                system_msg = system_msg_path.read()
            template.add_system_message(system_msg)

            # Load Prompt Template
            with open(prompt_path, "r") as prompt_file:
                prompt = prompt_file.read()
            template.add_user_message(prompt)
        ...

[dluc]

@joowon-dm-snu @tynguyen this is now implemented, pls see the latest nuget 0.12

example: https://github.com/microsoft/semantic-kernel/blob/main/samples/dotnet/kernel-syntax-examples/Example26_SemanticFunctionsUsingChatGPT.cs

IKernel kernel = new KernelBuilder().WithLogger(ConsoleLogger.Log).Build();

// Note: we use Chat Completion and GPT 3.5 Turbo
kernel.Config
    .AddAzureChatCompletionService("id", "gpt-35-turbo", "https://....openai.azure.com/", "...API KEY...");

var func = kernel.CreateSemanticFunction(
    "List the two planets closest to '{{$input}}', excluding moons, using bullet points.");

var result = await func.InvokeAsync("Jupiter");
Console.WriteLine(result);

[joowon-dm-snu]

@joowon-dm-snu @tynguyen this is now implemented, pls see the latest nuget 0.12

example: https://github.com/microsoft/semantic-kernel/blob/main/samples/dotnet/kernel-syntax-examples/Example26_SemanticFunctionsUsingChatGPT.cs

IKernel kernel = new KernelBuilder().WithLogger(ConsoleLogger.Log).Build();

// Note: we use Chat Completion and GPT 3.5 Turbo
kernel.Config
    .AddAzureChatCompletionService("id", "gpt-35-turbo", "https://....openai.azure.com/", "...API KEY...");

var func = kernel.CreateSemanticFunction(
    "List the two planets closest to '{{$input}}', excluding moons, using bullet points.");

var result = await func.InvokeAsync("Jupiter");
Console.WriteLine(result);

@dluc does your team has plan for python porting? if no one took it yet, I can take care. i need it

[awharrison-28]

@joowon-dm-snu ported, see https://github.com/microsoft/semantic-kernel/pull/643. Will be available with the next version of the pip package