gptel icon indicating copy to clipboard operation
gptel copied to clipboard

Gemini: parse code execution and grounding result

Open blahgeek opened this issue 7 months ago • 5 comments

(I'm not sure if this feature is desired or not. This is a draft version. Please let me know how you think.)

Gemini model support builtin tools including google search (grounding) and code execution. Now users can use :request-params to add support for them (https://github.com/karthink/gptel/issues/750#issuecomment-2815590261 ). This PR extends the parsing function for Gemini to print grounding and code execution results. (We may need better ways other than custom :request-params to support these tools, but that's for future PRs)

Screenshot 2025-04-27 at 21 55 21 Screenshot 2025-04-27 at 21 55 56

blahgeek avatar Apr 27 '25 14:04 blahgeek

@blahgeek Thanks for the PR. This feature is welcome. Can the codeExecution parsing be folded into the main cl-loop? Otherwise this looks fine.

karthink avatar Apr 29 '25 05:04 karthink

(We may need better ways other than custom :request-params to support these tools, but that's for future PRs)

It's best to expose these features as backend-specific tools, I think. There's no way in gptel or its UI to have tools that only work with one backend, so yeah, :request-params will do for now.

karthink avatar Apr 29 '25 05:04 karthink

Thanks. I've updated the code according to your suggestion.

blahgeek avatar Apr 29 '25 10:04 blahgeek

@blahgeek Nice! I think it would be interesting to have an option to extract the target redirected URL from the obfuscated "vertexaisearch.cloud.google.com" URLs. In Linux I've been doing this with curl -s -o /dev/null -w '%header{location}' --head <URL>.

Also, a couple of weeks ago I hacked together the insertion of grounding URLs with the associated citations throughout the LLM response (which shows where each grounding applies in the response). Unfortunately gptel's markdown->org conversion makes the :startIndex and :endIndex returned by Gemini incorrect for inserting the citations.

Here's the code, which may help in this PR:

https://gist.github.com/eslgastal/6700418bbad6ff9e49a4d6e28fe3dd83

And a screenshot:

gemini_grounding

eslgastal avatar May 03 '25 09:05 eslgastal

This looks great, are there any blockers to merging this?

avishefi avatar Aug 05 '25 11:08 avishefi