Citation has content but assistant return no data found

[robinzhengwang]

During the test, I found sometimes that the role "tool" has returned some citations within all related information. But the role "assistant" returned data not found. May I know how to deal with this issue? Is there something I can tune in OpenAI? Thank you.

[getpointful]

Similar issues here. Citations in a tool message but just the "not found" assistant message. Also: "not found" message with no citations, but plenty of highly scored hits in the data if you query the underlying cognitive search.

[psugasani]

I noticed that some of the issues lie with the intent generated by the model. Keeping the search service default with keyword search I found less accurate results. However, testing with semantic search I noticed the intent generated was more accurate and so the model was capable of generating an answer.

[lindazqli]

@robinzhengwang @getpointful Feel free to try out semantic/vector search and let us know if it helps. If not, feel free to open a support ticket with Azure OpenAI. Thanks :)

[robinzhengwang]

@robinzhengwang @getpointful Feel free to try out semantic/vector search and let us know if it helps. If not, feel free to open a support ticket with Azure OpenAI. Thanks :)

Thanks for reply. In this case, I am using default semantic search, but I got the same result. I have also tried some approaches to raise the search score (in this case I am using synonym mapping), but the "assistant" still return "Not Found". I also watched the intent difference continuously, I guess OpenAI cannot work well directly with synonym settings from Cognitive Search, it might be completed by prompt engineering. I will try to test it in Vector Search.

[robinzhengwang]

After I use vector search, and vectorSemanticHybrid query mode, I still found the issue that citation has content, but "assistant" still says not found.

[kdcllc]

@lindazqli and @sarah-widder,

I am not able to get the data for the first query for the following configurations:

AZURE_SEARCH_USE_SEMANTIC_SEARCH=false
AZURE_SEARCH_QUERY_TYPE=vectorSemanticHybrid
AZURE_SEARCH_SEMANTIC_SEARCH_CONFIG=content-search

I ended up adding the following code to the initial chat request:

    if len(request_messages) == 1 :
        if not any("Tell me more about AMETEK's businesses." in message['content'] for message in request_messages) :
            
            first_message = request_messages[0]
            first_message = request_messages.pop(0)

            request_messages.append({
                "role": "tool",
                "content": "{\"citations\": [], \"intent\": \"[]\"}",
                "end_turn": "true"
            })

            request_messages.append({
                "role": "assistant",
                "content": "Please modify your question. I am looking for information about a specific product, company, or service.",
                "end_turn": "true"
            })
            request_messages.append({
                "role": "assistant",
                "content": "return all possible products or services if an input only consist of alpha numeric value as a single word look for product number or company.",
                "end_turn": "true"
            })

            request_messages.append(first_message)

It would be helpful if the integration between Azure OpenAI and Azure AI Search would be open sourced.