Egon Elbre

Btw. @rahul2393 is there something I need to do to get this merged?

The issue does not seem to happen with GOOGLE_CLOUD_SPANNER_MULTIPLEXED_SESSIONS=false.

Only setting `GOOGLE_CLOUD_SPANNER_MULTIPLEXED_SESSIONS_FOR_RW=false` also seems to help.

still flaky https://build.dev.storj.tools/job/uplink-gerrit/3569/testReport/junit/storj/io_storj_satellite_admin/Build___Verification___Integration_using_Spanner__storj_storj____TestProjectDelete_withUsagePreviousMonthUncharged_Spanner/ ``` 7e4a-4dd0-b034-c604f03359a6", "queries": ""} project_test.go:1330: Error Trace: /go/pkg/mod/storj.io/[email protected]/satellite/admin/project_test.go:1330 /go/pkg/mod/storj.io/[email protected]/private/testplanet/run.go:76 /go/pkg/mod/storj.io/[email protected]/private/testmonkit/run.go:49 /go/pkg/mod/storj.io/[email protected]/private/testmonkit/run.go:43 /go/pkg/mod/storj.io/[email protected]/private/testplanet/run.go:54 Error: Not equal: expected: "{\"error\":\"usage for last month exist, but is not billed yet\",\"detail\":\"\"}" actual...

Note, these should also run when any of the console API changes.

Thnks for the suggestions; and indeed usually we fix such things with 2. approach.

Still flaky https://build.dev.storj.tools/job/storj-gerrit-verify/18438/testReport/junit/storj/io_storj_storagenode_multinode/TestPayoutsEndpointEstimations_Sqlite_EstimatedPayoutTotal/: ``` payout_test.go:206: Error Trace: /var/lib/jenkins/workspace/storj-gerrit-verify/storagenode/multinode/payout_test.go:206 Error: Not equal: expected: 246850 actual : 246849 Test: TestPayoutsEndpointEstimations/Sqlite/EstimatedPayoutTotal --- FAIL: TestPayoutsEndpointEstimations/Sqlite/EstimatedPayoutTotal (0.01s) ```

Flaky https://build.dev.storj.tools/job/storj-gerrit-verify/25554/#showFailuresLink ``` gc_test.go:253: Error Trace: /var/lib/jenkins/workspace/storj-gerrit-verify/satellite/gc/gc_test.go:253 /var/lib/jenkins/workspace/storj-gerrit-verify/private/testplanet/run.go:80 /var/lib/jenkins/workspace/storj-gerrit-verify/private/testmonkit/run.go:49 /var/lib/jenkins/workspace/storj-gerrit-verify/private/testmonkit/run.go:43 /var/lib/jenkins/workspace/storj-gerrit-verify/private/testplanet/run.go:58 Error: Not equal: expected: 65792 actual : 66048 Test: TestGarbageCollectionWithCopies/SyncObserverV2/Spanner ```

Looks like this is also flaky in uplink https://build.dev.storj.tools/job/uplink-gerrit/3788/testReport/junit/storj/io_storj_satellite_metainfo/Build___Integration_storj_storj_using_Spanner___TestRateLimit_ProjectRateLimitOverrideCachedExpired_Spanner/: ``` endpoint_test.go:747: Error Trace: /go/pkg/mod/storj.io/[email protected]/satellite/metainfo/endpoint_test.go:747 /go/pkg/mod/storj.io/[email protected]/private/testplanet/run.go:80 /go/pkg/mod/storj.io/[email protected]/private/testmonkit/run.go:49 /go/pkg/mod/storj.io/[email protected]/private/testmonkit/run.go:43 /go/pkg/mod/storj.io/[email protected]/private/testplanet/run.go:58 Error: "[]" should have 1 item(s), but has 0 Test: TestRateLimit_ProjectRateLimitOverrideCachedExpired/Spanner ```

Used to happen with CRDB as well https://github.com/storj/storj/issues/5714