[Feature Request]: Support HTTPS proxies and self-signed certificates for GCS staging

[Abacn]

What would you like to happen?

Creating dataflow pipelines via Proxy Server with own java trust store will encounter certificate error:

WARNING: exception thrown while executing request
java.net.UnknownHostException: www.googleapis.com
                at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
                at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
                at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
                at java.net.Socket.connect(Socket.java:589)
                at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673)
                at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
                at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
                at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
                at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
                at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
                at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
                at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
                at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
                at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
                at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162)
                at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:93)

this is due to the hardcoded google vendored trust store in java-api-library. To support customer supplied certificates (mTLS) it needs to upgrade google-api-client dependency to 1.38.0 and expose it in beam sdk.

Issue Priority

Priority: 2

Issue Component

Component: io-java-gcp

[Abacn]

Currently the dependency google_cloud_platform_libraries_bom ver 25.2.0 already sets google-http-client version to be 1.41.7 which should support customer supplied certificates. Need to expose this or possibly upgrade gcs client.