semantic-kernel icon indicating copy to clipboard operation
semantic-kernel copied to clipboard

Problems with the `is_native` function in functions_view.py

Open salmon131 opened this issue 2 years ago • 0 comments

Describe the bug There is a problem with FunctionsView's native function validation.

def is_semantic(self, skill_name: str, function_name: str) -> bool:
    as_sf = self._semantic_functions.get(skill_name, [])
    as_sf = any(f.name == function_name for f in as_sf)

    as_nf = self._native_functions.get(skill_name, [])
    as_nf = any(f.name == function_name for f in as_nf)

    if as_sf and as_nf:
        raise KernelException(
            KernelException.ErrorCodes.AmbiguousImplementation,
            f"There are 2 functions with the same name: {function_name}."
            f"One is native and the other semantic.",
        )

    return as_sf

def is_native(self, skill_name: str, function_name: str) -> bool:
    return not self.is_semantic(skill_name, function_name)

The result we want to achieve by using that method should look like this.

as_sf as_nf is_semantic Result is_native Result
True True KernelException KernelException
True False True False
False True False True
False False False False

However, with the code above, the result is as follows: the is_native function will return True even if as_nf is False, i.e., the given native function was not checked.

as_sf as_nf is_semantic Result is_native Result
True True KernelException KernelException
True False True False
False True False True
False False False True

To Reproduce Here is my test code.

Prepare in advance

import semantic_kernel as sk
from semantic_kernel.core_skills import MathSkill
import semantic_kernel.connectors.ai.open_ai as sk_oai

kernel = sk.Kernel()

model = "text-davinci-002"
service_id = model

# Configure AI service used by the kernel
api_key, org_id = sk.openai_settings_from_dot_env()
kernel.add_text_completion_service(
    service_id, sk_oai.OpenAITextCompletion(model, api_key, org_id)
)

# Import Semantic and Native skill
kernel.import_semantic_skill_from_directory("../samples/skills", "FunSkill")
kernel.import_skill(MathSkill(), "math")

# Verify that the skill has been imported
print(kernel.skills.get_functions_view()._semantic_functions)
print(kernel.skills.get_functions_view()._native_functions)

Output

{'FunSkill': [<semantic_kernel.skill_definition.function_view.FunctionView object at 0x7f8ba39068b0>, <semantic_kernel.skill_definition.function_view.FunctionView object at 0x7f8ba39067f0>, <semantic_kernel.skill_definition.function_view.FunctionView object at 0x7f8ba3906820>]}
{'math': [<semantic_kernel.skill_definition.function_view.FunctionView object at 0x7f8ba39067f0>, <semantic_kernel.skill_definition.function_view.FunctionView object at 0x7f8ba3906820>]}

Normal Case

# Determine if the skill is semantic or native
print(kernel.skills.get_functions_view().is_semantic("FunSkill", "Joke"))
print(kernel.skills.get_functions_view().is_native("math", "Add"))

In this case, both the semantic and native functions exist within the given skill, so it outputs True, True.

True
True

Problem Case

# Determine if the skill is semantic or native
print(kernel.skills.get_functions_view().is_semantic("FunSkill", "Joke"))
print(kernel.skills.get_functions_view().is_native("math", "Missing Function"))

In this case, is_native is returning True even though function not in math(which is natvie skill).

True
True

Expected behavior Therefore, the native function check logic should be modified as follows:

def is_semantic(self, skill_name: str, function_name: str) -> bool:
    as_sf = self._semantic_functions.get(skill_name, [])
    as_sf = any(f.name == function_name for f in as_sf)

    as_nf = self._native_functions.get(skill_name, [])
    as_nf = any(f.name == function_name for f in as_nf)

    if as_sf and as_nf:
        raise KernelException(
            KernelException.ErrorCodes.AmbiguousImplementation,
            f"There are 2 functions with the same name: {function_name}."
            f"One is native and the other semantic.",
        )

    return as_sf

def is_native(self, skill_name: str, function_name: str) -> bool:
    as_sf = self._semantic_functions.get(skill_name, [])
    as_sf = any(f.name == function_name for f in as_sf)

    as_nf = self._native_functions.get(skill_name, [])
    as_nf = any(f.name == function_name for f in as_nf)

    if as_sf and as_nf:
        raise KernelException(
            KernelException.ErrorCodes.AmbiguousImplementation,
            f"There are 2 functions with the same name: {function_name}."
            f"One is native and the other semantic.",
        )

    return as_nf

Screenshots image

Desktop (please complete the following information):

  • OS: macOS
  • IDE: VSCode
  • NuGet Package Version [e.g. 0.1.0]

Additional context Except, I had one question. In _normalize_names in the skill_collection, it performs lower_case on skill_name and function_name to check for case insensitive skill and function names. However, inside the functions_view, it performs an exact match without normalizing. I think these two processes are inconsistent. Shouldn't the same process be applied in functions_view? Please comment if you agree.

salmon131 avatar May 25 '23 05:05 salmon131