Questions about gemini support in the spring ai module

[yangfeng20]

I want to use the gemini apiKey to access my gemini Ai instead of using project-id+location.

Currently, the spring ai module using gemini only supports project-id+location using google cloud. The apiKey mode in Google ai studio is not supported.

My request was to support the apiKey pattern, similar to openai.

[tzolov]

Hey @yangfeng20 could you provide us with a the Gemini documentation that explains how to create and use api keys?

[yangfeng20]

Ok, please refer to: https://ai.google.dev/gemini-api/docs/api-overview https://ai.google.dev/gemini-api/docs/api-key https://ai.google.dev/gemini-api/docs/get-started/rest

[vanduc2514]

@yangfeng20 There is a way to work around with the current spring-ai-vertex-ai-gemini dependency by replacing the low-level Request Formatter in HttpJsonPredictionServiceStub with the one pointing to the api endpoint of new Gemini AI

ProtoMessageRequestFormatter.<GenerateContentRequest>newBuilder()
                .setPath("/v1beta/models/{model=*}:generateContent", request -> Map.of("model", "gemini-pro"))
                .setQueryParamsExtractor(request -> Map.of("key", List.of("<GEMINI_API_KEY>")))
                .setRequestBodyExtractor(request -> ProtoRestSerializer.create().toBody("*", request.toBuilder().clearModel().build(),false))
                .build()

then set the VertexAI endpoint to https://generativelanguage.googleapis.com and use Transport.REST for VertexAI transport. The above code snippet only works for generateContent endpoint. For other endpoints to work, you will need to replace other request formatters. It's ok for the time being since Google has not provided an offical Java depdency yet.

[marcosamm]

@yangfeng20 There is a way to work around with the current spring-ai-vertex-ai-gemini dependency by replacing the low-level Request Formatter in HttpJsonPredictionServiceStub with the one pointing to the api endpoint of new Gemini AI

ProtoMessageRequestFormatter.<GenerateContentRequest>newBuilder()
                .setPath("/v1beta/models/{model=*}:generateContent", request -> Map.of("model", "gemini-pro"))
                .setQueryParamsExtractor(request -> Map.of("key", List.of("<GEMINI_API_KEY>")))
                .setRequestBodyExtractor(request -> ProtoRestSerializer.create().toBody("*", request.toBuilder().clearModel().build(),false))
                .build()

then set the VertexAI endpoint to https://generativelanguage.googleapis.com and use Transport.REST for VertexAI transport. The above code snippet only works for generateContent endpoint. For other endpoints to work, you will need to replace other request formatters. It's ok for the time being since Google has not provided an offical Java depdency yet.

@vanduc2514 , I didn't understand. Where do I need to set the transport to REST? Could you provide a short functional example using ProtoMessageRequestFormatter and setting Transport.REST?

[vanduc2514]

@marcosamm Sorry for making not clear. The new version of Spring AI 1.0.0-SNAPSHOT does not require the Transport.REST to be set. To implement this work-around you will need to create a custom implementation of PredictionServiceStub usingProtoMessageRequestFormatterthat points to the Gemini endpoint. Then use it for the VertexAiGeminiChatModel.

Here is an example for how to do it

https://gist.github.com/vanduc2514/0a74ec0ec160538c39313e73f27f8740

Then you will use the VertexAIAdapter class from the gist instead of the default VertexAI

@Bean
ChatModel chatModel() {
    var vertexAIAdapter = new VertexAIAdapter("gemini-base-url", "gemini-api-key")
    return new VertexAiGeminiChatModel(vertexAIAdapter);
}

You also need to disable VertexAiGeminiAutoConfiguration for skipping the auto bean creation.

[marcosamm]

@vanduc2514 , this works for me.

Sometimes I get timeout exceptions like this:

java.net.SocketTimeoutException: Read timed out
	at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:278) ~[na:na]
	at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304) ~[na:na]
	at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346) ~[na:na]
	at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796) ~[na:na]
	at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1099) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1461) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1066) ~[na:na]
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:291) ~[na:na]
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347) ~[na:na]
	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420) ~[na:na]
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:399) ~[na:na]
	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:827) ~[na:na]
	at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:759) ~[na:na]
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1690) ~[na:na]
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1599) ~[na:na]
	at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531) ~[na:na]
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:307) ~[na:na]
	at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36) ~[google-http-client-1.44.2.jar:1.44.2]
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:152) ~[google-http-client-1.44.2.jar:1.44.2]
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[google-http-client-1.44.2.jar:1.44.2]
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[google-http-client-1.44.2.jar:1.44.2]
	at com.google.api.gax.httpjson.HttpRequestRunnable.run(HttpRequestRunnable.java:115) ~[gax-httpjson-2.49.0.jar:2.49.0]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[na:na]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

But it's normal in more complex interactions.

Thank you very much!

[vanduc2514]

@marcosamm You are welcome, I'm glad that it works for you. The issue you have seems related to SSL handshake timeout, does it frequently happen ?

[markpollack]

Hi. I spoke with the google team, this support is supposed to be coming in a newer version of their Java SDK. I'd rather wait for them to provide this than introduce workarounds. I'm frustrated as well as it prevents us from creating integration tests on github actions.

[yangfeng20]

Thanks for your reply, which means that the gemini api scheduling will be implemented in spring ai using google's sdk. Thank you for your previous efforts. Can you release the code you wrote earlier as a third-party project? I want to learn from it. Thank you.

[marcosamm]

@marcosamm You are welcome, I'm glad that it works for you. The issue you have seems related to SSL handshake timeout, does it frequently happen ?

Sorry for the delay. Yes, it happens frequently, especially when many tokens are sent.

[account123456789]

@vanduc2514 , this works for me.

Sometimes I get timeout exceptions like this:

java.net.SocketTimeoutException: Read timed out
	at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:278) ~[na:na]
	at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304) ~[na:na]
	at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346) ~[na:na]
	at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796) ~[na:na]
	at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1099) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1461) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1066) ~[na:na]
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:291) ~[na:na]
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347) ~[na:na]
	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420) ~[na:na]
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:399) ~[na:na]
	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:827) ~[na:na]
	at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:759) ~[na:na]
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1690) ~[na:na]
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1599) ~[na:na]
	at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531) ~[na:na]
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:307) ~[na:na]
	at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36) ~[google-http-client-1.44.2.jar:1.44.2]
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:152) ~[google-http-client-1.44.2.jar:1.44.2]
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[google-http-client-1.44.2.jar:1.44.2]
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[google-http-client-1.44.2.jar:1.44.2]
	at com.google.api.gax.httpjson.HttpRequestRunnable.run(HttpRequestRunnable.java:115) ~[gax-httpjson-2.49.0.jar:2.49.0]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[na:na]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

But it's normal in more complex interactions.

Thank you very much!

Any Solution for the timeout issue , bro ?? I am facing it too much with Gemini unfortunately.