storage: specifying grpc.WithTransportCredentials default to storage.mtls.googleapis.com endpoint

[salrashid123]

Storage gRPC client automatically defaults to mtls endpoint if the transportcredentials is set

In the follwoing code, if i want to just specify my own transport credentials (so that i can keylog), the endpoint defaults to the mtls endpoint (which i dont' want here).

If you run the follwoing with the env vars set

	   export GRPC_GO_LOG_SEVERITY_LEVEL=info
	   export GRPC_GO_LOG_VERBOSITY_LEVEL=99
	   export GODEBUG=http2debug=2

you'll see it attempts to connect to the mtls endpoint though i didn't specify it

2025/03/11 12:00:46 INFO: [core] Creating new client transport to "{Addr: \"storage.mtls.googleapis.com:443\", ServerName: \"storage.mtls.googleapis.com:443\", }": connection error: desc = "error reading server preface: remote error: tls: certificate required"
2025/03/11 12:00:46 WARNING: [core] [Channel #1 SubChannel #2]grpc: addrConn.createTransport failed to connect to {Addr: "storage.mtls.googleapis.com:443", ServerName: "storage.mtls.googleapis.com:443", }. Err: connection error: desc = "error reading server preface: remote error: tls: certificate required"
2025/03/11 12:00:46 INFO: [core] [Channel #1 SubChannel #2]Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "error reading server preface: remote error: tls: certificate required"

the code below works if you uncomment the to not specify transportcredentials or if you specify the endpoint directly.

---

package main

/*
 */
import (
	"crypto/tls"
	"flag"
	"fmt"
	"io"
	"log"
	"os"
	"time"

	"cloud.google.com/go/storage"
	"golang.org/x/net/context"
	"google.golang.org/api/option"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials"
)

const ()

var (
	bucketName = flag.String("bucketName", "someproject", "Bucket to upload/download")
	fileName   = flag.String("fileName", "foo.txt", "Bucket to upload/download")
)

func main() {

	flag.Parse()

	ctx, cancel := context.WithTimeout(context.Background(), time.Second*10)
	defer cancel()

	/*
	   export GRPC_GO_LOG_SEVERITY_LEVEL=info
	   export GRPC_GO_LOG_VERBOSITY_LEVEL=99
	   export GODEBUG=http2debug=2
	*/
	tlsCfg := &tls.Config{
		MinVersion: tls.VersionTLS13,
	}

	sslKeyLogfile := os.Getenv("SSLKEYLOGFILE")
	if sslKeyLogfile != "" {
		var w *os.File
		w, err := os.OpenFile(sslKeyLogfile, os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0600)
		if err != nil {
			log.Fatalf("Could not create keylogger: %v", err)
		}
		tlsCfg.KeyLogWriter = w
	}
	rpcreds := credentials.NewTLS(tlsCfg)

	storageClient, err := storage.NewGRPCClient(ctx, option.WithGRPCDialOption(grpc.WithTransportCredentials(rpcreds)))

	//storageClient, err := storage.NewGRPCClient(ctx, option.WithEndpoint("storage.googleapis.com:443"), option.WithGRPCDialOption(grpc.WithTransportCredentials(rpcreds)))
	//storageClient, err := storage.NewGRPCClient(ctx)
	if err != nil {
		fmt.Printf("storage.NewClient: %v", err)
		return
	}
	defer storageClient.Close()

	bkt := storageClient.Bucket(*bucketName)

	rc, err := bkt.Object(*fileName).NewReader(ctx)
	if err != nil {
		fmt.Printf("Error Reading: Object(%q).NewReader: %v", *fileName, err)
		return
	}
	defer rc.Close()

	if _, err := io.Copy(os.Stdout, rc); err != nil {
		fmt.Printf("io.Copy: %v", err)
		return
	}
	fmt.Println("")

}

go.mod

module main

go 1.24.0

require (
	cloud.google.com/go/storage v1.50.0
	golang.org/x/net v0.37.0
	google.golang.org/api v0.224.0
	google.golang.org/grpc v1.71.0
)

[tritone]

@codyoss @quartzmo Do you have any idea about this? I think whatever logic is picking the mTLS endpoint is happening at the auth layer; I don't see us explicitly setting it in storage.

@salrashid123 can you describe what problem occurs when the mTLS endpoint is picked?

[salrashid123]

@tritone if the mtls endpoint is selected and the client certs aren't provided, the connection itself fails

[codyoss]

We don't have a good way of detecting what options are passed into gRPC. If you wish to disable the use of mTLS you can set GOOGLE_API_USE_MTLS_ENDPOINT to never as documented here: https://google.aip.dev/auth/4114. Does that help?

[salrashid123]

yeah, it thnk thats a workaround after the fact though.

i mean if i run the code as-is with the tlsconfig as shown, all i'd see is this

$ go run main.go 

Error Reading: Object("foo.txt").NewReader: retry failed with context deadline exceeded; last error:

 rpc error: code = Unavailable desc = connection error: 
    desc = "error reading server preface: remote error: tls: certificate required"

and i woud have to guess that its due to the mtls endpoint thats selected and subsequently to turn it off with that flag. While "certificate required" error hints at something to do with mtls, its a stretch to know what the issue is and apply the workaround (though this issue itself can be cited for the fix)