serpapi
[Google AI Overview API] Snippet References Missing and Wrong Object
In the ai_overview object we are seeing text blocks that have references but no reference_indexes and also responses with snippet_links and no reference_indexes.
We are seeing AI Overview objects be returned with an object that should only be coming from AI Mode's endpoint snippet_links. We are attempting to map the references to the correct snippets in the AIO and because the reference indexes are either missing (seemingly being replaced by snippet_links) or include references that don't show up in the reference object--see screenshot for a snippet with 4 reference indexes, but only 1 total reference was returned for the entire AIO.
Search 1 - Snippet Links: https://serpapi.com/searches/68d18c34aa0a88faea22e5b8/inspect Search 2 - Missing References: https://serpapi.com/searches/68d18c37bd1c20f286cbb255/inspect
Thanks for reaching out, @jaceiverson. If you're referring to the actual AI Overview response, can you share the search ids of these AI Overview calls, as the ones you shared are only Google Search API calls that do not include the AI Overview, but rather the page_token to retrieve it in the following call:
If you're referring to other AI Overview mentions, there are some related_questions of the ai_overview type, but I can't find anything unusual in these. In https://serpapi.com/searches/68d18c34aa0a88faea22e5b8/inspect, the snippet_links look to be reasonable if they are indeed included in the snippet. They are not limited to the Google AI Mode API and can also be included in the related_questions as in this example:
And in https://serpapi.com/searches/68d18c37bd1c20f286cbb255/inspect, I can't see missing references:
Again, these are taken from related_questions, so they might not be what you're referring to. If that's the case, please share the direct AI Overview calls so I can investigate them further, as the ones you shared do not include the AI Overview information you're potentially referring to.
@jaceiverson, I've been able to recreate the issue with missing reference_indexes, and reported it here: https://github.com/serpapi/public-roadmap/issues/3052. I don't see an issue with snippet_links appearing in the AI Overview response, and it looks to me like they did not replace the index_references, and the fact that the latter are missing is an independent issue and not related to the snippet_links.
I couldn't reproduce the issue with references missing, so can you please share an example of that?
@marm123 Thanks for the speedy response. Let me clarify, when I said missing references, I mean that we have reference indexes that exist, in this example 0, 1, 4, 5 but only a single reference in the ai_overview section of the JSON. Here is the full JSON response: https://gist.github.com/jaceiverson/014e7b742299fd3f067257e445eb8db4. So its almost the inverse of the issue that you created on #3052. That issue there are references but no reference_indexes. Here there are more reference_indexes but no references.
Thanks @jaceiverson. Just to clarify, was the AI Overview included in https://serpapi.com/searches/68d18c37bd1c20f286cbb255/inspect, or did you retrieve it from a different call and concatenated the Google Search API response with it? I'm asking because it's no longer available in https://serpapi.com/searches/68d18c37bd1c20f286cbb255/inspect, which might be because the AI Overview is short-lived, but I want to confirm I'm not missing anything.
The AIO was retrieved using a second call. The full object ai_overview has been replaced in the original response. So that gist I linked above has the original response for everything but the ai_overview is there. And ai_overview object is the data from the second call.
Can you share the ID of that call @jaceiverson? I'm currently unable to recreate it, and all the additional information associated with the call (not only the response itself) can be very helpful in further analysis.
@marm123 I'm sorry I don't have the ID of that call. I am going to update our logs to make sure to output that value from the AIO extra call. Let me find another example for you.
Thank you @jaceiverson. Yes, if you could find the id of the search where this happens, that would be great. I tried to recreate an issue with your query, but was unsuccessful as all the references were correctly returned (both yesterday and today).
@jaceiverson Just a heads up that we fixed the issue with missing reference_indexes. Noted in this other issue: https://github.com/serpapi/public-roadmap/issues/3052
