VectorStoreClient sometimes returns "I could not find an answer"

[shhlife]

• [ x] I have read and agree to the contributing guidelines.

Describe the bug When using the VectorStoreClient, it sometimes responds with "I could not find an answer" even when the answer should be able to be returned.

To Reproduce

I created a relatively simple agent that takes a data file and upserts it, then asks some questions. It should be able to figure it out, but it doesn't.

Here's the text file: lunchtime.md

# Lunch
* monday: fried chicken
* tuesday: pork
* wednesday: vegetable sandwiches
* thursday: also pork
* friday: dumplings, with pork

# Dinner
* We don't serve dinner. Go home!

And here's the script

from dotenv import load_dotenv
from pathlib import Path

from griptape.structures import Agent
from griptape.tools import VectorStoreClient, TaskMemoryClient
from griptape.loaders import TextLoader
from griptape.engines import VectorQueryEngine
from griptape.drivers import LocalVectorStoreDriver, OpenAiEmbeddingDriver


load_dotenv()

namespace = "lunchtime"

# Create an engine
engine = VectorQueryEngine(
    vector_store_driver=LocalVectorStoreDriver(embedding_driver=OpenAiEmbeddingDriver())
)

engine.upsert_text_artifacts(
    TextLoader().load(Path("lunchtime.md")), namespace=namespace
)

vector_db = VectorStoreClient(
    description="This DB has helpful information",
    query_engine=engine,
    namespace=namespace,
    off_prompt=False,
)


agent = Agent(tools=[vector_db], stream=True)

agent.run("What's for dinner monday?")
agent.run("What's for dinner?")

When you run it, when you ask "What's for dinner Monday" it responds with I could not find an answer. But if you ask "What's for dinner" it responds with the correct We don't serve dinner, go home.

Expected behavior I would expect it to return We don't serve dinner, go home with the first question as well as the second.

Additional info I also found if I asked "When are we having food with pork?" it would reply with "We are having food with pork on Tuesday and Thursday", but it would forget about Friday.

[mgcollie]

Here was the original thread that sparked his issue: https://discord.com/channels/1096466116672487547/1140674107600551987/1200638078134517800

[qwe321389yfs8t7huqd89q]

What is your priority on solving this issue?

[collindutter]

This seems to be a combination of the phrasing of We don't serve dinner. Go home! and the VectorQueryEngine's default model of gpt-3.5-turbo. The model is getting confused getting confused.

I would suggest either updating the VectorQueryEngine's model to gpt-4 or updating the Dinner description a bit.

Here is a working version for me:

from dotenv import load_dotenv
from pathlib import Path

from griptape.structures import Agent
from griptape.tools import VectorStoreClient
from griptape.loaders import TextLoader
from griptape.engines import VectorQueryEngine
from griptape.drivers import (
    LocalVectorStoreDriver,
    OpenAiChatPromptDriver,
    OpenAiEmbeddingDriver,
)


load_dotenv()

namespace = "lunchtime"

# Create an engine
engine = VectorQueryEngine(
    prompt_driver=OpenAiChatPromptDriver(model="gpt-4"),
    vector_store_driver=LocalVectorStoreDriver(
        embedding_driver=OpenAiEmbeddingDriver()
    ),
)

engine.upsert_text_artifacts(
    TextLoader().load(Path("lunchtime.md")), namespace=namespace
)

vector_db = VectorStoreClient(
    description="This DB has helpful information",
    query_engine=engine,
    namespace=namespace,
    off_prompt=False,
)


agent = Agent(tools=[vector_db])

agent.run("What's for dinner monday?")
agent.run("What's for dinner?")

[02/13/24 11:02:19] INFO     ToolkitTask ec897030219f4047aed091c335d7fa5f
                             Input: What's for dinner monday?
[02/13/24 11:02:25] INFO     Subtask d54162d7187c48c99fba04a424de5821
                             Thought: To find out what's for dinner on Monday, I need to search the vector database with the query "What's for dinner Monday?".
                             Action: {"name": "VectorStoreClient", "path": "search", "input": {"values": {"query": "What's for dinner Monday?"}}}
[02/13/24 11:02:26] INFO     Subtask d54162d7187c48c99fba04a424de5821
                             Response: We don't serve dinner. Go home!
[02/13/24 11:02:27] INFO     ToolkitTask ec897030219f4047aed091c335d7fa5f
                             Output: There's no dinner served on Monday.
                    INFO     ToolkitTask ec897030219f4047aed091c335d7fa5f
                             Input: What's for dinner?
[02/13/24 11:02:44] INFO     Subtask 892657ab770649f383319e14519528f9
                             Thought: I need to search the vector database to find out what's for dinner. I'll use the VectorStoreClient action to do this.

                             Action:
                             {
                               "name": "VectorStoreClient",
                               "path": "search",
                               "input": {
                                 "values": {
                                   "query": "What's for dinner?"
                                 }
                               }
                             }
[02/13/24 11:02:45] INFO     Subtask 892657ab770649f383319e14519528f9
                             Response: We don't serve dinner. Go home!
[02/13/24 11:02:47] INFO     ToolkitTask ec897030219f4047aed091c335d7fa5f
                             Output: We don't serve dinner. Go home!

[mgcollie]

@collindutter from my testing this is still a problem and the fix only works in simple cases but the proposed solution doesn't work for more complicated examples that spawned the reason why this issue was opened. I think this issue was closed perhaps prematurely.

[mgcollie]

shotgun-api-examples.md

using the attached file, and this simple example:

from dotenv import load_dotenv
from pathlib import Path
import logging

from griptape.structures import Agent
from griptape.utils import Chat
from griptape.tools import VectorStoreClient
from griptape.drivers import (
    LocalVectorStoreDriver,
    OpenAiEmbeddingDriver,
    OpenAiChatPromptDriver
)
from griptape.engines import VectorQueryEngine
from griptape.loaders import TextLoader
from griptape.rules import Rule, Ruleset


load_dotenv()
# Upsert documentation  into the vector database
namespace = "shotgrid_api"

if __name__ == '__main__':
    # Create the query engine
    engine = VectorQueryEngine(
        vector_store_driver=LocalVectorStoreDriver(embedding_driver=OpenAiEmbeddingDriver()),
        prompt_driver=OpenAiChatPromptDriver(model="gpt-4"))

    # Load the API documentation
    examples = TextLoader().load(Path(r'shotgun-api-examples.md'))
    engine.upsert_text_artifacts(examples, namespace)

    # Instantiate the Vector Store Client
    vector_store_tool = VectorStoreClient(
        description="Contains information about ShotGrid api. Use it to help with ShotGrid client requests.",
        query_engine=engine,
        namespace=namespace,
        off_prompt=False,
    )

    # Create the ruleset
    shotgrid_agent_ruleset = Ruleset(
        name="ShotGrid Agent",
        rules=[
            Rule("Only Use the Vectorstore Client to answer questions about the shotgrid api"),

        ],
    )

    # Instantiate the agent
    agent = Agent(
        logger_level=logging.ERROR,
        tools=[vector_store_tool],
        rulesets=[shotgrid_agent_ruleset],
        stream=True
    )

    # Start chatting
    Chat(agent).start()

if you ask the question "How to find active projects that are not archived in ShotGrid?" you will reliably get an answer that tells you you need to use the following filter syntax filters = [ ['sg_status', 'is', 'Googleputz'], # Adjust this value based on your ShotGrid setup ['archived', 'is', False] ]

[mgcollie]

however if you use something more complicated that has other tools, more rules, and more information in the VectorStoreClient then the results become more unreliable

from dotenv import load_dotenv
from textwrap import dedent
from pathlib import Path
import os
import logging

from griptape.structures import Agent
from griptape.utils import Chat
from griptape.tools import DateTime, VectorStoreClient, Calculator
from griptape.drivers import (
    LocalVectorStoreDriver,
    OpenAiEmbeddingDriver,
    OpenAiChatPromptDriver
)
from griptape.engines import VectorQueryEngine
from griptape.loaders import WebLoader, TextLoader
from griptape.rules import Rule, Ruleset

from shotgrid_tool import ShotGridTool

load_dotenv()
# Upsert documentation  into the vector database
namespace = "shotgrid_api"


if __name__ == '__main__':
    # Create the query engine
    engine = VectorQueryEngine(
        vector_store_driver=LocalVectorStoreDriver(embedding_driver=OpenAiEmbeddingDriver()),
        prompt_driver=OpenAiChatPromptDriver(model="gpt-4"))

    # API Documentation
    shotgrid_api_urls = [
        "https://developers.shotgridsoftware.com/python-api/reference.html",
        "https://developers.shotgridsoftware.com/python-api/reference.html#filter-syntax",
        "https://developers.shotgridsoftware.com/python-api/cookbook/usage_tips.html",
        "https://developers.shotgridsoftware.com/python-api/cookbook/attachments.html",
        "https://developers.shotgridsoftware.com/python-api/cookbook/tasks/updating_tasks.html",
        "https://developers.shotgridsoftware.com/python-api/cookbook/tasks/task_dependencies.html",
        "https://developers.shotgridsoftware.com/python-api/cookbook/tasks/split_tasks.html",
    ]

    # Load the API documentation
    examples = TextLoader().load(Path(r'shotgun-api-examples.md'))
    engine.upsert_text_artifacts(examples, namespace)

    for url in shotgrid_api_urls:
        print(url)
        artifacts = WebLoader().load(url)
        engine.upsert_text_artifacts(artifacts, namespace)

    # Instantiate the Vector Store Client
    vector_store_tool = VectorStoreClient(
        description="Contains information about ShotGrid api. Use it to help with ShotGrid client requests.",
        query_engine=engine,
        namespace=namespace,
        off_prompt=False,
    )

    SHOTGRID_URL = os.getenv("SHOTGRID_URL")
    SHOTGRID_API_KEY = os.getenv("SHOTGRID_API_KEY")
    SHOTGRID_SCRIPT = "GriptapeApi"
    SHOTGRID_USER = os.getenv("SHOTGRID_USER")
    SHOTGRID_PASSWORD = os.getenv("SHOTGRID_PASSWORD")

    # Instantiate the tool
    shotgrid_tool = ShotGridTool(
        base_url=SHOTGRID_URL,
        api_key=SHOTGRID_API_KEY,
        script_name=SHOTGRID_SCRIPT,
        user_login=SHOTGRID_USER,
        user_password=SHOTGRID_PASSWORD,
        login_method="user",
        off_prompt=False
    )

    # Create the ruleset
    shotgrid_agent_ruleset = Ruleset(
        name="ShotGrid Agent",
        rules=[
            Rule("Act as a studio coordinator who is an expert with Autodesk ShotGrid"),
            Rule(
                "Your only objective is to find and update information in ShotGrid using the ShotGridTool"
            ),
            Rule(
                dedent(
                    """
                For specific information about how to use ShotGrid API activities, the 
                VectorStoreClient should be used. Take the necessary time to consult the 
                VectorStoreClient to ensure the most accurate and context-aware decisions 
                when interacting with the ShotGridTool and API.
                """
                )
            ),
            Rule(
                dedent(
                    """
                Always use the ShotGrid batch() method when performing multiple create, update, or delete operations in 
                ShotGrid. This should be done in a single request to improve efficiency and speed. Batch method uses 
                "request_type", "entity_type", and "entity_id". Failure to use the batch() method in these instances will 
                be considered a violation of the rules."""
                )
            ),
            Rule(
                dedent(
                    """
                When asked to find, list, create, update, delete, retrieve details of entities, 
                update task status, update task data, list task assignments, retrieve the history of changes to 
                entities, manage versions of assets, manage project timelines, track project 
                progress, manage notes and reviews, manage user roles and permissions, or 
                integrate with other tools and workflows, the ShotGrid Client API is always used."""
                )
            ),
        ],
    )

    # Instantiate the agent
    agent = Agent(
        logger_level=logging.ERROR,
        tools=[DateTime(off_prompt=False), Calculator(off_prompt=False), shotgrid_tool, vector_store_tool],
        rulesets=[shotgrid_agent_ruleset],
        stream=True
    )

    # Start chatting
    Chat(agent).start()

[mgcollie]

@collindutter & @shhlife to test the results I've been running the application, asking the question, shutting it down, running it again, asking the same question, and repeating this process.

[collindutter]

Thanks for the extra context @mgcollie, and apologies for the premature closing! Will continue to play with this to see if we can reproduce and resolve for good.

[collindutter]

@mgcollie apologies for never following back up with this. Quite a bit has changed with regards to RAG/Vector Stores since this original issue.

I'm going to close this, but if you're still encountering this and are able to provide a minimal reproducible example, I'd be happy to dig into it again.