[Bug]: Agent almost never generates answer despite data is available in datastore.

[jeromemassot]

File Name

tutorial_vertex_ai_search_rag_agent.ipynb

What happened?

For all the questions proposed to the deployed agent in this notebook, the agent replies : "I cannot fulfill this request. The available tools do not have information about ..." despite the queried information is present in the structure data.

Relevant log output

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[koverholt]

Thanks for opening this issue. I attempted to reproduce but was not able to, the notebook is working as expected for me and is returning movies based on records in the dataset.

Ensure that you followed the tutorial steps here: https://cloud.google.com/generative-ai-app-builder/docs/try-enterprise-search#structured-data

Ensure that the search app is working as expected when you preview the search app and search for a sample query: https://cloud.google.com/generative-ai-app-builder/docs/try-enterprise-search#preview_your_app

Ensure that the code cell that tests the Vertex AI Search retriever function is working as expected:

search_kaggle_movies("Harry Potter")

And if you continue to run into errors, can you post more details about:

• The environment that you're running in
• The versions of google-cloud-aiplatform, langchain, langchain-core, langchain-google-community, and other relevant LangChain and Google Cloud client libs
• Full error messages w/ stack traces and intermediate steps
• Model versions that you're using
• Whether local agent and remote agent is working, or details if only one is working

[jeromemassot]

Hi Kristopher, thanks for your comment. I confirm that the Datastore is consistent with the Kaggle dataset. And if I query the agent 10 times I may have sometimes an answer close to the expected output. However, most of the time, the agent is not capable to generate the answer despite the retrieved context seems ok (tested directly in the Agent Builder). So I confirm that the overall performance of the agent is very low in my environment despite a consistent datastore. Best Regards Jerome

PS: the remote agent has same issue. Deployed using the given code in the notebook

remote_agent = reasoning_engines.ReasoningEngine.create(
    agent,
    requirements=[
        "google-cloud-aiplatform[langchain,reasoningengine]",
        "cloudpickle==3.0.0",
        "pydantic==2.7.4",
        "langchain-google-community",
        "google-cloud-discoveryengine",
    ],
)

[koverholt]

Thanks for the info that the local agent is having the same issue. In order for me to reproduce, can you post more details about:

• The environment that you're running in
• The versions of google-cloud-aiplatform, langchain, langchain-core, langchain-google-community, and other relevant LangChain and Google Cloud client libs
• Full error messages w/ stack traces and intermediate steps
• Model versions that you're using
• Full list of steps that you took to create and index data in Vertex AI Search

[koverholt]

Closing due to inactivity. Please reopen if you have an update here. Thanks!