google-cloud-go
google-cloud-go copied to clipboard
googleapis
bigtable/bttest: default credentials error despite using insecure connection
Client
Bigtable
Go Environment
go 1.22.1
Code
e.g.
server, err := bttest.NewServer("localhost:0")
if err != nil {
t.Fatalf("start bigtable emulator server: %v", err)
}
t.Cleanup(server.Close)
conn, err := grpc.NewClient(server.Addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
t.Fatalf("connect to bigtable emulator instance: %v", err)
}
adminClient, err := bigtable.NewAdminClient(ctx, "emulator-test", "instance", option.WithGRPCConn(conn))
if err != nil {
t.Fatalf("create admin client: %v", err)
}
Expected behavior
No error, client is successfully created.
Actual behavior
It works until bigtable v1.25.0, but since v1.27.1 I get an error:
create admin client: credentials: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
Additional context
Same happens with bigtable.NewClient, is not specific to the admin client.
I do not know which specific version between 1.25.0 and 1.27.1 broke it.
@jakobatfreiheit could you please share your go.mod. I wrote the following program with the follow go.mod and have no issues:
package main
import (
"context"
"log/slog"
"os"
"strings"
"cloud.google.com/go/bigtable"
"cloud.google.com/go/bigtable/bttest"
"google.golang.org/api/option"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
func main() {
if err := run(); err != nil {
slog.Error("shutting down", "err", err)
os.Exit(1)
}
}
func run() error {
ctx := context.Background()
server, err := bttest.NewServer("localhost:0")
if err != nil {
return err
}
defer server.Close()
conn, err := grpc.NewClient(server.Addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
return err
}
adminClient, err := bigtable.NewAdminClient(ctx, "emulator-test", "instance", option.WithGRPCConn(conn))
if err != nil {
return err
}
defer adminClient.Close()
resp, err := adminClient.Tables(ctx)
if err != nil {
return err
}
slog.Info(strings.Join(resp, ","))
return nil
}
go.mod
module example.com/dialconn
go 1.23.0
require (
cloud.google.com/go/bigtable v1.27.1
google.golang.org/api v0.189.0
google.golang.org/grpc v1.64.1
)
require (
cel.dev/expr v0.15.0 // indirect
cloud.google.com/go v0.115.0 // indirect
cloud.google.com/go/auth v0.7.2 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.3 // indirect
cloud.google.com/go/compute/metadata v0.5.0 // indirect
cloud.google.com/go/iam v1.1.10 // indirect
cloud.google.com/go/longrunning v0.5.9 // indirect
cloud.google.com/go/monitoring v1.20.1 // indirect
github.com/census-instrumentation/opencensus-proto v0.4.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cncf/xds/go v0.0.0-20240423153145-555b57ec207b // indirect
github.com/envoyproxy/go-control-plane v0.12.0 // indirect
github.com/envoyproxy/protoc-gen-validate v1.0.4 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/google/btree v1.1.2 // indirect
github.com/google/s2a-go v0.1.7 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
github.com/googleapis/gax-go/v2 v2.13.0 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
go.opentelemetry.io/otel v1.24.0 // indirect
go.opentelemetry.io/otel/metric v1.24.0 // indirect
go.opentelemetry.io/otel/sdk v1.24.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.24.0 // indirect
go.opentelemetry.io/otel/trace v1.24.0 // indirect
golang.org/x/crypto v0.25.0 // indirect
golang.org/x/net v0.27.0 // indirect
golang.org/x/oauth2 v0.21.0 // indirect
golang.org/x/sync v0.7.0 // indirect
golang.org/x/sys v0.22.0 // indirect
golang.org/x/text v0.16.0 // indirect
golang.org/x/time v0.5.0 // indirect
google.golang.org/genproto v0.0.0-20240722135656-d784300faade // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240722135656-d784300faade // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240722135656-d784300faade // indirect
google.golang.org/protobuf v1.34.2 // indirect
rsc.io/binaryregexp v0.2.0 // indirect
)
Sorry, I just realized I had made a mistake in my original message:
The problem does not happen with bigtable.NewAdminClient but with bigtable.NewClient.
So for example running this in a Go docker image (where no credentials are set, I used this one: https://hub.docker.com/r/arm64v8/golang/) reproduces the problem:
package main
import (
"context"
"log/slog"
"os"
"cloud.google.com/go/bigtable"
"cloud.google.com/go/bigtable/bttest"
"google.golang.org/api/option"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
func main() {
if err := run(); err != nil {
slog.Error("shutting down", "err", err)
os.Exit(1)
}
}
func run() error {
ctx := context.Background()
server, err := bttest.NewServer("localhost:0")
if err != nil {
return err
}
defer server.Close()
conn, err := grpc.NewClient(server.Addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
return err
}
client, err := bigtable.NewClient(ctx, "emulator-test", "instance", option.WithGRPCConn(conn))
if err != nil {
return err
}
defer client.Close()
slog.Info("Hi")
return nil
}
This is the go.mod:
module example
go 1.23.0
require (
cloud.google.com/go/bigtable v1.29.0
google.golang.org/api v0.192.0
google.golang.org/grpc v1.65.0
)
require (
cel.dev/expr v0.15.0 // indirect
cloud.google.com/go v0.115.0 // indirect
cloud.google.com/go/auth v0.8.1 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.3 // indirect
cloud.google.com/go/compute/metadata v0.5.0 // indirect
cloud.google.com/go/iam v1.1.12 // indirect
cloud.google.com/go/longrunning v0.5.11 // indirect
cloud.google.com/go/monitoring v1.20.3 // indirect
github.com/census-instrumentation/opencensus-proto v0.4.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cncf/xds/go v0.0.0-20240423153145-555b57ec207b // indirect
github.com/envoyproxy/go-control-plane v0.12.0 // indirect
github.com/envoyproxy/protoc-gen-validate v1.0.4 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/google/btree v1.1.2 // indirect
github.com/google/s2a-go v0.1.8 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
github.com/googleapis/gax-go/v2 v2.13.0 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
go.opentelemetry.io/otel v1.24.0 // indirect
go.opentelemetry.io/otel/metric v1.24.0 // indirect
go.opentelemetry.io/otel/sdk v1.24.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.24.0 // indirect
go.opentelemetry.io/otel/trace v1.24.0 // indirect
golang.org/x/crypto v0.25.0 // indirect
golang.org/x/net v0.27.0 // indirect
golang.org/x/oauth2 v0.22.0 // indirect
golang.org/x/sync v0.8.0 // indirect
golang.org/x/sys v0.22.0 // indirect
golang.org/x/text v0.16.0 // indirect
golang.org/x/time v0.6.0 // indirect
google.golang.org/genproto v0.0.0-20240730163845-b1a4ccb954bf // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240725223205-93522f1f2a9f // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240730163845-b1a4ccb954bf // indirect
google.golang.org/protobuf v1.34.2 // indirect
rsc.io/binaryregexp v0.2.0 // indirect
)
I added the following dockerfile, updated to use your provided code, and ran it:
FROM golang
COPY . /app/
WORKDIR /app
CMD ["go", "run", "main.go"]
Output:
2024/08/19 14:10:00 INFO Hi
I am still unable to reproduce the issue.
Actually I think I see the issue. I believe this was introduced with #10046. @bhshkh it looks like we need the metrics provider stuff to be more of a no-op in some cases. This flow tries to dial instead of doing nothings and/or using the provided connection.
I believe https://github.com/googleapis/google-cloud-go/pull/10658 should fix this
Any update on when we will likely see the fix merged? We're still stuck on v1.25, we have not found a quick-fix to work around this issue.
https://github.com/googleapis/google-cloud-go/pull/10815 has the fix bigtable 1.32.0
Tested with this code:
package main
import (
"context"
"testing"
"cloud.google.com/go/bigtable"
"cloud.google.com/go/bigtable/bttest"
"google.golang.org/api/option"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
func TestIssue10688(t *testing.T) {
server, err := bttest.NewServer("localhost:0")
if err != nil {
t.Fatalf("start bigtable emulator server: %v", err)
}
t.Cleanup(server.Close)
conn, err := grpc.NewClient(server.Addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
t.Fatalf("connect to bigtable emulator instance: %v", err)
}
ctx := context.Background()
_, err = bigtable.NewAdminClient(ctx, "emulator-test", "instance", option.WithGRPCConn(conn))
if err != nil {
t.Fatalf("create admin client: %v", err)
}
}
$ go test -race -v -timeout 45m ./...
=== RUN TestIssue10688
--- PASS: TestIssue10688 (0.01s)
PASS
ok issue-10688 1.572s