google-cloud-go
google-cloud-go copied to clipboard
googleapis
bigtable: error logs writing metrics after upgrade from v1.39.0 to v1.40.1
Client
Bigtable
Environment
Container-optimised OS with containerd running in GKE Go 1.25
Our code is running in one GCP project, and the Bigtable instance is in another.
Code and Dependencies
You really don't want to see all our code!
go.mod
module github.com/unravelin/core
go 1.25.0
tool (
github.com/mailru/easyjson/easyjson
github.com/philpearl/ut/genmock
github.com/valyala/quicktemplate/qtc
golang.org/x/tools/cmd/stringer
mvdan.cc/gofumpt
)
replace github.com/unravelin/null => github.com/unravelin/null/v5 v5.0.1
replace github.com/dvsekhvalnov/jose2go v1.5.0 => github.com/unravelin/jose2go v1.5.1-0.20211018104955-81c3a93a988f
replace github.com/modern-go/reflect2 => github.com/unravelin/reflect2 v1.0.4
replace github.com/json-iterator/go => github.com/unravelin/jsoniter v1.1.17
replace github.com/nsqio/go-nsq => github.com/unravelin/go-nsq v1.1.1
require (
cloud.google.com/go v0.123.0
cloud.google.com/go/artifactregistry v1.17.2
cloud.google.com/go/bigquery v1.71.0
cloud.google.com/go/bigtable v1.40.1
cloud.google.com/go/cloudbuild v1.23.1
cloud.google.com/go/compute/metadata v0.9.0
cloud.google.com/go/container v1.44.1
cloud.google.com/go/iam v1.5.3
cloud.google.com/go/logging v1.13.0
cloud.google.com/go/profiler v0.4.3
cloud.google.com/go/pubsub/v2 v2.3.0
cloud.google.com/go/recaptchaenterprise/v2 v2.20.5
cloud.google.com/go/storage v1.57.0
github.com/DataDog/datadog-go v4.8.3+incompatible
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/trace v1.30.0
github.com/PuerkitoBio/goquery v1.10.3
github.com/Vivino/go-autocomplete-trie v0.0.0-20230301121706-da951497d081
github.com/avct/uasurfer v0.0.0-20240501094946-ca0c4d1e541b
github.com/bitly/go-hostpool v0.1.1
github.com/blang/semver v3.5.1+incompatible
github.com/bradfitz/latlong v0.0.0-20170410180902-f3db6d0dff40
github.com/braintree-go/braintree-go v0.22.0
github.com/buger/jsonparser v1.1.1
github.com/ctessum/macreader v0.0.0-20151005232333-d4ef8112dbfa
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
github.com/davegardnerisme/phonegeocode v0.0.0-20160120101024-a49b977f8889
github.com/dnaeon/go-vcr v1.2.0
github.com/extrame/xls v0.0.0-20160702142058-da346a5ffb8b
github.com/getkin/kin-openapi v0.133.0
github.com/go-co-op/gocron/v2 v2.17.0
github.com/go-zookeeper/zk v1.0.4
github.com/golang-jwt/jwt/v5 v5.3.0
github.com/google/go-cmp v0.7.0
github.com/google/go-replayers/grpcreplay v1.3.0
github.com/google/gofuzz v1.2.0
github.com/google/uuid v1.6.0
github.com/googleapis/gax-go/v2 v2.15.0
github.com/gorilla/mux v1.8.1
github.com/grafana-tools/sdk v0.0.0-20220919052116-6562121319fc
github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/icio/mkcert v0.1.2
github.com/infobloxopen/go-trees v0.0.0-20221216143356-66ceba885ebc
github.com/invopop/jsonschema v0.13.0
github.com/invopop/yaml v0.3.1
github.com/jackc/pgx/v5 v5.7.6
github.com/jonas-p/go-shp v0.1.1
github.com/jonomacd/jordered v0.0.0-20220111163942-e0826e67aea4
github.com/json-iterator/go v1.1.12
github.com/julienschmidt/httprouter v1.3.0
github.com/justinas/nosurf v1.2.0
github.com/keighl/postmark v0.0.0-20180713155648-e30e577cc7fb
github.com/klauspost/compress v1.18.1
github.com/klauspost/pgzip v1.2.6
github.com/kr/pretty v0.3.1
github.com/lib/pq v1.10.9
github.com/mailru/easyjson v0.9.0
github.com/mmcloughlin/geohash v0.10.0
github.com/modern-go/reflect2 v1.0.2
github.com/nsqio/go-nsq v1.1.0
github.com/nyaruka/phonenumbers v1.6.6
github.com/paulmach/orb v0.12.0
github.com/peterh/liner v1.2.2
github.com/philpearl/avro v0.0.34
github.com/philpearl/csv v0.0.4
github.com/philpearl/intern v0.0.1
github.com/philpearl/levenshtein v0.0.0-20170212162545-0dbaccfeda92
github.com/philpearl/mmap v0.0.1
github.com/philpearl/plenc v0.0.22
github.com/philpearl/scratchbuild v0.0.2-0.20230504084801-d24c04a25f26
github.com/philpearl/symboltab v1.1.4
github.com/philpearl/symboltab/offheap v1.0.8
github.com/philpearl/ut v0.0.5
github.com/pquerna/otp v1.5.0
github.com/prometheus/client_golang v1.23.2
github.com/pusher/pusher-http-go/v5 v5.1.1
github.com/rainycape/unidecode v0.0.0-20150907023854-cb7f23ec59be
github.com/redis/go-redis/extra/rediscmd/v9 v9.12.1
github.com/redis/go-redis/v9 v9.14.0
github.com/rgeoghegan/tabulate v0.0.0-20170527210506-f65c88667bb4
github.com/robfig/cron/v3 v3.0.1
github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e
github.com/slack-go/slack v0.17.3
github.com/soheilhy/cmux v0.1.5
github.com/spf13/cast v1.10.0
github.com/spf13/cobra v1.10.1
github.com/stretchr/testify v1.11.1
github.com/stripe/stripe-go/v72 v72.122.0
github.com/stripe/stripe-go/v81 v81.4.0
github.com/svix/svix-webhooks v1.77.0
github.com/thejerf/slogassert v0.3.4
github.com/tidwall/gjson v1.18.0
github.com/tidwall/jsonc v0.3.2
github.com/tidwall/sjson v1.2.5
github.com/tsenart/vegeta v12.7.0+incompatible
github.com/tsenart/vegeta/v12 v12.12.0
github.com/ua-parser/uap-go v0.0.0-20210121150957-347a3497cc39
github.com/unravelin/go-jose/v2 v2.6.3
github.com/unravelin/gorm v0.0.0-20240328132654-655c9ba72e76
github.com/unravelin/null v4.1.0+incompatible
github.com/unravelin/pci v0.0.0-20251104161840-f9f7bc33f9a5
github.com/unravelin/ravelinml v0.0.0-20250610170712-41548c0d1f84
github.com/valyala/fasthttp v1.67.0
github.com/valyala/quicktemplate v1.8.0
github.com/xeipuuv/gojsonschema v1.2.0
github.com/zalando/go-keyring v0.2.6
github.com/zeebo/xxh3 v1.0.2
go.opentelemetry.io/contrib/detectors/gcp v1.38.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0
go.opentelemetry.io/otel v1.38.0
go.opentelemetry.io/otel/sdk v1.38.0
go.opentelemetry.io/otel/trace v1.38.0
golang.org/x/crypto v0.43.0
golang.org/x/mod v0.29.0
golang.org/x/net v0.46.0
golang.org/x/oauth2 v0.32.0
golang.org/x/sync v0.17.0
golang.org/x/sys v0.37.0
golang.org/x/term v0.36.0
golang.org/x/text v0.30.0
golang.org/x/time v0.14.0
golang.org/x/tools v0.38.0
google.golang.org/api v0.252.0
google.golang.org/genproto/googleapis/rpc v0.0.0-20251002232023-7c0ddcbb5797
google.golang.org/grpc v1.76.0
google.golang.org/protobuf v1.36.10
gopkg.in/go-playground/validator.v9 v9.31.0
gopkg.in/yaml.v3 v3.0.1
k8s.io/api v0.30.6
k8s.io/apimachinery v0.30.6
k8s.io/client-go v0.30.6
mvdan.cc/gofumpt v0.9.1
)
require (
al.essio.dev/pkg/shellescape v1.5.1 // indirect
cel.dev/expr v0.24.0 // indirect
cloud.google.com/go/auth v0.17.0 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
cloud.google.com/go/longrunning v0.6.7 // indirect
cloud.google.com/go/monitoring v1.24.2 // indirect
cloud.google.com/go/trace v1.11.6 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.29.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.53.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.54.0 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/andybalholm/brotli v1.2.0 // indirect
github.com/andybalholm/cascadia v1.3.3 // indirect
github.com/apache/arrow/go/v15 v15.0.2 // indirect
github.com/arbovm/levenshtein v0.0.0-20160628152529-48b4e1c0c4d0 // indirect
github.com/bahlo/generic-list-go v0.2.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cncf/xds/go v0.0.0-20250501225837-2ac532fd4443 // indirect
github.com/danieljoos/wincred v1.2.2 // indirect
github.com/denisenkom/go-mssqldb v0.0.0-20191124224453-732737034ffd // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
github.com/envoyproxy/go-control-plane/envoy v1.32.4 // indirect
github.com/envoyproxy/protoc-gen-validate v1.2.1 // indirect
github.com/erikstmartin/go-testdb v0.0.0-20160219214506-8d10e4a1bae5 // indirect
github.com/evanphx/json-patch v5.6.0+incompatible // indirect
github.com/extrame/ole2 v0.0.0-20160623023314-ebc59eca74fc // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-jose/go-jose/v4 v4.1.2 // indirect
github.com/go-json-experiment/json v0.0.0-20250910080747-cc2cfa0554c3 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-openapi/jsonpointer v0.21.1 // indirect
github.com/go-openapi/jsonreference v0.20.4 // indirect
github.com/go-openapi/swag v0.23.1 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-sql-driver/mysql v1.4.1 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/godbus/dbus/v5 v5.1.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/golang/snappy v1.0.0 // indirect
github.com/google/flatbuffers v25.2.10+incompatible // indirect
github.com/google/gnostic-models v0.6.8 // indirect
github.com/google/pprof v0.0.0-20250602020802-c6617b811d0e // indirect
github.com/google/s2a-go v0.1.9 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.6 // indirect
github.com/gorilla/websocket v1.5.3 // indirect
github.com/gosimple/slug v1.13.1 // indirect
github.com/gosimple/unidecode v1.0.1 // indirect
github.com/imdario/mergo v0.3.16 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/influxdata/tdigest v0.0.2-0.20210216194612-fc98d27c9e8b // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/jinzhu/gorm v1.9.12 // indirect
github.com/jinzhu/now v1.1.1 // indirect
github.com/jonboulle/clockwork v0.5.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/klauspost/cpuid/v2 v2.2.5 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/mattn/go-runewidth v0.0.3 // indirect
github.com/mattn/go-sqlite3 v2.0.3+incompatible // indirect
github.com/moby/spdystream v0.2.0 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f // indirect
github.com/oasdiff/yaml v0.0.0-20250309154309-f31be36b4037 // indirect
github.com/oasdiff/yaml3 v0.0.0-20250309153720-d2182401db90 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/perimeterx/marshmallow v1.1.5 // indirect
github.com/philpearl/stringbank v1.2.0 // indirect
github.com/philpearl/stringbank/offheap v1.0.3 // indirect
github.com/pierrec/lz4/v4 v4.1.18 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.66.1 // indirect
github.com/prometheus/procfs v0.16.1 // indirect
github.com/qor/inflection v0.0.0-20160626222713-8f4d3a0d04ce // indirect
github.com/rogpeppe/go-internal v1.14.1 // indirect
github.com/rs/dnscache v0.0.0-20230804202142-fc85eb664529 // indirect
github.com/satori/go.uuid v1.2.0 // indirect
github.com/spf13/pflag v1.0.9 // indirect
github.com/spiffe/go-spiffe/v2 v2.5.0 // indirect
github.com/tchap/go-patricia v2.2.6+incompatible // indirect
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.1 // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/wk8/go-ordered-map/v2 v2.1.8 // indirect
github.com/woodsbury/decimal128 v1.3.0 // indirect
github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f // indirect
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
github.com/zeebo/errs v1.4.0 // indirect
go.mongodb.org/mongo-driver v1.13.1 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.63.0 // indirect
go.opentelemetry.io/otel/metric v1.38.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.38.0 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect
goji.io v2.0.2+incompatible // indirect
golang.org/x/exp v0.0.0-20250408133849-7e4ce0ab07d0 // indirect
golang.org/x/image v0.18.0 // indirect
golang.org/x/telemetry v0.0.0-20251008203120-078029d740a8 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/genproto v0.0.0-20250922171735-9219d122eba9 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250922171735-9219d122eba9 // indirect
gopkg.in/go-playground/assert.v1 v1.2.1 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
k8s.io/klog/v2 v2.120.1 // indirect
k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 // indirect
k8s.io/utils v0.0.0-20230726121419-3b25d923346b // indirect
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect
sigs.k8s.io/yaml v1.3.0 // indirect
)
Actual & Expected behavior
Our code is running in one GCP project (call it p1), and the Bigtable instance is running in another (call it p2). Prior to upgrading our code was writing Bigtable client-side metrics to p2 without issue.
After the upgrade we saw errors logged about permissions to write metrics to p1. We granted that permission and saw more frequent errors like the following. We do not believe our code is logging these errors - we expect it must be Bigtable libraries or their dependencies.
rpc error: code = InvalidArgument desc = Field resource.labels.project_id had an invalid value of "p2": if present, must be the project number or ID in the request name (projects/p1)
The metrics that were previously written to p2 without issue are still written there without issue, so presumably these logs are about new metrics that are being written to p1.
We would expect the metrics to ideally be written without error to project p2 where the other metrics are logged and the Bigtable instance is running. We would be reasonably happy if the metrics were logged to p1 without error. We would be moderately happy if we could keep the old Bigtable client-side metrics and turn off the new metrics - looking at the code there seems no way to do this.
Additional context
Started after upgrading from v1.39.0 to v1.40.1
This should be fixed in https://github.com/googleapis/google-cloud-go/pull/13286. We will release a patch in 1.40.2
@philpearl @adamroyjones the fix should be rolled out in 1.41.0. we did not release a patch.
Also, we would like you to try Direct Access on golang. https://docs.google.com/document/d/1Auuo-cSTDExkOoQS29ac7euLCM5vK2w8lRdKpbbn6GA/edit?pli=1&tab=t.0
If you are running in GKE/GCE with no special firewall, setting an env var CBT_ENABLE_DIRECTPATH=true should do it. More details on doc. Happy to answer question if necessary.
@sushanb: Thanks for the update and for pointing us to Bigtable Direct Access. It looks very promising!
We're currently in a code freeze but we'll upgrade to v1.41.0 and look into enabling direct access from 5 January (when things will thaw).
Reduced P99 latency? Just take my money now. Oh, you already are! Anyway, that sounds great. Long tail latencies are our biggest problem with Bigtable and have been for years so its great to see some improvement there
I've noticed the CheckDirectAccessSupported call checks that the CBT_ENABLE_DIRECTPATH env var is set to true. So you can't check if the underlying network is OK for direct access without enabling direct access for general use. It would be good to remove these checks from this function if possible
@philpearl we are working on making this better see https://github.com/sushanb/google-cloud-go/pull/4. This will be released in 1.42.0.
Basiclaly, this skips all manual checks for direct access . I will be in touch once the client is ready for production use.
thanks. happy new year in advance.