relayer icon indicating copy to clipboard operation
relayer copied to clipboard

Published 20 hours ago • verified publisher icon cosmos

[Bug] intermittent race conditions in CI on `make interchaintest-events`

Open jtieri opened this issue 1 year ago • 4 comments

Noticed that there is an intermittent race condition detected in CI when running make interchaintest-events. I have not been able to reproduce this issue locally no matter how many runs I have attempted.

Here are some logs:

==================
WARNING: DATA RACE
Write at 0x00c0018c60c0 by goroutine 1027:
  runtime.mapassign_faststr()
      /opt/hostedtoolcache/go/1.21.3/x64/src/runtime/map_faststr.go:203 +0x0
  github.com/cosmos/cosmos-sdk/types.(*Config).SetBech32PrefixForAccount()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/types/config.go:89 +0x13d
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SetSDKContext()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/bech32_hack.go:19 +0x2f
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).buildMessages()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:605 +0xbd
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessagesToMempool()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:175 +0x1cf
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessages.func2()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:128 +0x144
  github.com/avast/retry-go/v4.Do.func1()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:115 +0x2e
  github.com/avast/retry-go/v4.DoWithData[go.shape.interface {}]()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:179 +0xb0f
  github.com/avast/retry-go/v4.Do()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:118 +0x6b
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessages()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:127 +0x46c
  github.com/cosmos/relayer/v2/relayer.CreateClient.func3()
      /home/runner/work/relayer/relayer/relayer/client.go:218 +0xe3
  github.com/avast/retry-go/v4.Do.func1()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:115 +0x2e
  github.com/avast/retry-go/v4.DoWithData[go.shape.interface {}]()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:179 +0xb0f
  github.com/avast/retry-go/v4.Do()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:118 +0x6b
  github.com/cosmos/relayer/v2/relayer.CreateClient()
      /home/runner/work/relayer/relayer/relayer/client.go:215 +0x1370
  github.com/cosmos/relayer/v2/relayer.(*Chain).CreateClients.func4()
      /home/runner/work/relayer/relayer/relayer/client.go:66 +0x190
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x76

Previous read at 0x00c0018c60c0 by goroutine 23:
  runtime.mapaccess1_faststr()
      /opt/hostedtoolcache/go/1.21.3/x64/src/runtime/map_faststr.go:13 +0x0
  github.com/cosmos/cosmos-sdk/types.(*Config).GetBech32AccountAddrPrefix()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/types/config.go:162 +0x7d
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewDefaultSigningOptions()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:94 +0x15c
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewTxConfigWithOptions()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:191 +0x[544](https://github.com/cosmos/relayer/actions/runs/6870999378/job/18687001943?pr=1334#step:5:545)
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewTxConfig()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:79 +0x127
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.MakeCodecConfig()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/codec.go:98 +0xe7
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.MakeCodec()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/codec.go:71 +0x1c4
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.CosmosProviderConfig.NewProvider()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/provider.go:123 +0x31a
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProviderConfig).NewProvider()
      <autogenerated>:1 +0x108
  github.com/cosmos/relayer/v2/cmd.(*ConfigInputWrapper).RuntimeConfig()
      /home/runner/work/relayer/relayer/cmd/config.go:337 +0x305
  github.com/cosmos/relayer/v2/cmd.(*appState).loadConfigFile()
      /home/runner/work/relayer/relayer/cmd/appstate.go:84 +0x391
  github.com/cosmos/relayer/v2/cmd.NewRootCmd.func1()
      /home/runner/work/relayer/relayer/cmd/root.go:77 +0x64
  github.com/spf13/cobra.(*Command).execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:954 +0xd25
  github.com/spf13/cobra.(*Command).ExecuteC()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x604
  github.com/spf13/cobra.(*Command).Execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039 +0x324
  github.com/spf13/cobra.(*Command).ExecuteContext()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1032 +0x31c
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunWithInputC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:83 +0x30e
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:57 +0x3d1
  github.com/cosmos/relayer/v2/interchaintest.(*Relayer).RestoreKey()
      /home/runner/work/relayer/relayer/interchaintest/relayer.go:106 +0x25
  github.com/strangelove-ventures/interchaintest/v8.(*Interchain).configureRelayerKeys()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/interchain.go:431 +0x54f
  github.com/strangelove-ventures/interchaintest/v8.(*Interchain).Build()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/interchain.go:263 +0xaa4
  github.com/strangelove-ventures/interchaintest/v8.StartChainPair()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/test_setup.go:73 +0x817
  github.com/strangelove-ventures/interchaintest/v8/conformance.TestChainPair()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:331 +0x891
  github.com/strangelove-ventures/interchaintest/v8/conformance.Test.func1.1.1.2()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:257 +0x259
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44

Goroutine 1027 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x124
  github.com/cosmos/relayer/v2/relayer.(*Chain).CreateClients()
      /home/runner/work/relayer/relayer/relayer/client.go:63 +0x9d4
  github.com/cosmos/relayer/v2/cmd.createClientsCmd.func1()
      /home/runner/work/relayer/relayer/cmd/tx.go:102 +0x61c
  github.com/spf13/cobra.(*Command).execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0xf2f
  github.com/spf13/cobra.(*Command).ExecuteC()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x604
  github.com/spf13/cobra.(*Command).Execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039 +0x324
  github.com/spf13/cobra.(*Command).ExecuteContext()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1032 +0x31c
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunWithInputC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:83 +0x30e
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:57 +0x2f1
  github.com/cosmos/relayer/v2/interchaintest.(*Relayer).CreateClients()
      /home/runner/work/relayer/relayer/interchaintest/relayer.go:240 +0x2d
  github.com/strangelove-ventures/interchaintest/v8/conformance.TestRelayerSetup.func2()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/relayersetup.go:82 +0x20a
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44

Goroutine 23 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x82a
  github.com/strangelove-ventures/interchaintest/v8/conformance.Test.func1.1.1()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:247 +0x31d
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44
==================
==================
WARNING: DATA RACE
Write at 0x00c0018c2988 by goroutine 1027:
  github.com/cosmos/cosmos-sdk/types.(*Config).SetBech32PrefixForAccount()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/types/config.go:89 +0x14c
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SetSDKContext()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/bech32_hack.go:19 +0x2f
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).buildMessages()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:605 +0xbd
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessagesToMempool()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:175 +0x1cf
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessages.func2()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:128 +0x144
  github.com/avast/retry-go/v4.Do.func1()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:115 +0x2e
  github.com/avast/retry-go/v4.DoWithData[go.shape.interface {}]()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:179 +0xb0f
  github.com/avast/retry-go/v4.Do()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:118 +0x6b
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessages()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:127 +0x46c
  github.com/cosmos/relayer/v2/relayer.CreateClient.func3()
      /home/runner/work/relayer/relayer/relayer/client.go:218 +0xe3
  github.com/avast/retry-go/v4.Do.func1()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:115 +0x2e
  github.com/avast/retry-go/v4.DoWithData[go.shape.interface {}]()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:179 +0xb0f
  github.com/avast/retry-go/v4.Do()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:118 +0x6b
  github.com/cosmos/relayer/v2/relayer.CreateClient()
      /home/runner/work/relayer/relayer/relayer/client.go:215 +0x1370
  github.com/cosmos/relayer/v2/relayer.(*Chain).CreateClients.func4()
      /home/runner/work/relayer/relayer/relayer/client.go:66 +0x190
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x76

Previous read at 0x00c0018c2988 by goroutine 23:
  github.com/cosmos/cosmos-sdk/types.(*Config).GetBech32AccountAddrPrefix()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/types/config.go:162 +0x87
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewDefaultSigningOptions()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:94 +0x15c
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewTxConfigWithOptions()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:191 +0x544
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewTxConfig()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:79 +0x127
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.MakeCodecConfig()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/codec.go:98 +0xe7
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.MakeCodec()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/codec.go:71 +0x1c4
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.CosmosProviderConfig.NewProvider()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/provider.go:123 +0x31a
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProviderConfig).NewProvider()
      <autogenerated>:1 +0x108
  github.com/cosmos/relayer/v2/cmd.(*ConfigInputWrapper).RuntimeConfig()
      /home/runner/work/relayer/relayer/cmd/config.go:337 +0x305
  github.com/cosmos/relayer/v2/cmd.(*appState).loadConfigFile()
      /home/runner/work/relayer/relayer/cmd/appstate.go:84 +0x391
  github.com/cosmos/relayer/v2/cmd.NewRootCmd.func1()
      /home/runner/work/relayer/relayer/cmd/root.go:77 +0x64
  github.com/spf13/cobra.(*Command).execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:954 +0xd25
  github.com/spf13/cobra.(*Command).ExecuteC()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x604
  github.com/spf13/cobra.(*Command).Execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039 +0x324
  github.com/spf13/cobra.(*Command).ExecuteContext()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1032 +0x31c
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunWithInputC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:83 +0x30e
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:57 +0x3d1
  github.com/cosmos/relayer/v2/interchaintest.(*Relayer).RestoreKey()
      /home/runner/work/relayer/relayer/interchaintest/relayer.go:106 +0x25
  github.com/strangelove-ventures/interchaintest/v8.(*Interchain).configureRelayerKeys()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/interchain.go:431 +0x54f
  github.com/strangelove-ventures/interchaintest/v8.(*Interchain).Build()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/interchain.go:263 +0xaa4
  github.com/strangelove-ventures/interchaintest/v8.StartChainPair()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/test_setup.go:73 +0x817
  github.com/strangelove-ventures/interchaintest/v8/conformance.TestChainPair()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:331 +0x891
  github.com/strangelove-ventures/interchaintest/v8/conformance.Test.func1.1.1.2()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:257 +0x259
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44

Goroutine 1027 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x124
  github.com/cosmos/relayer/v2/relayer.(*Chain).CreateClients()
      /home/runner/work/relayer/relayer/relayer/client.go:63 +0x9d4
  github.com/cosmos/relayer/v2/cmd.createClientsCmd.func1()
      /home/runner/work/relayer/relayer/cmd/tx.go:102 +0x61c
  github.com/spf13/cobra.(*Command).execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0xf2f
  github.com/spf13/cobra.(*Command).ExecuteC()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x604
  github.com/spf13/cobra.(*Command).Execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039 +0x324
  github.com/spf13/cobra.(*Command).ExecuteContext()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1032 +0x31c
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunWithInputC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:83 +0x30e
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:57 +0x2f1
  github.com/cosmos/relayer/v2/interchaintest.(*Relayer).CreateClients()
      /home/runner/work/relayer/relayer/interchaintest/relayer.go:240 +0x2d
  github.com/strangelove-ventures/interchaintest/v8/conformance.TestRelayerSetup.func2()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/relayersetup.go:82 +0x20a
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44

Goroutine 23 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x82a
  github.com/strangelove-ventures/interchaintest/v8/conformance.Test.func1.1.1()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:247 +0x31d
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44
==================
==================
WARNING: DATA RACE
Write at 0x00c0018c60c0 by goroutine 1028:
  runtime.mapassign_faststr()
      /opt/hostedtoolcache/go/1.21.3/x64/src/runtime/map_faststr.go:203 +0x0
  github.com/cosmos/cosmos-sdk/types.(*Config).SetBech32PrefixForAccount()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/types/config.go:89 +0x13d
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SetSDKContext()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/bech32_hack.go:19 +0x2f
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).buildMessages()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:605 +0xbd
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessagesToMempool()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:175 +0x1cf
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessages.func2()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:128 +0x144
  github.com/avast/retry-go/v4.Do.func1()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:115 +0x2e
  github.com/avast/retry-go/v4.DoWithData[go.shape.interface {}]()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:179 +0xb0f
  github.com/avast/retry-go/v4.Do()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:118 +0x6b
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessages()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:127 +0x46c
  github.com/cosmos/relayer/v2/relayer.CreateClient.func3()
      /home/runner/work/relayer/relayer/relayer/client.go:218 +0xe3
  github.com/avast/retry-go/v4.Do.func1()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:115 +0x2e
  github.com/avast/retry-go/v4.DoWithData[go.shape.interface {}]()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:179 +0xb0f
  github.com/avast/retry-go/v4.Do()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:118 +0x6b
  github.com/cosmos/relayer/v2/relayer.CreateClient()
      /home/runner/work/relayer/relayer/relayer/client.go:215 +0x1370
  github.com/cosmos/relayer/v2/relayer.(*Chain).CreateClients.func5()
      /home/runner/work/relayer/relayer/relayer/client.go:76 +0x190
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x76

Previous read at 0x00c0018c60c0 by goroutine 23:
  runtime.mapaccess1_faststr()
      /opt/hostedtoolcache/go/1.21.3/x64/src/runtime/map_faststr.go:13 +0x0
  github.com/cosmos/cosmos-sdk/types.(*Config).GetBech32AccountAddrPrefix()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/types/config.go:162 +0x7d
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewDefaultSigningOptions()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:94 +0x15c
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewTxConfigWithOptions()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:191 +0x544
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewTxConfig()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:79 +0x127
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.MakeCodecConfig()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/codec.go:98 +0xe7
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.MakeCodec()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/codec.go:71 +0x1c4
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.CosmosProviderConfig.NewProvider()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/provider.go:123 +0x31a
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProviderConfig).NewProvider()
      <autogenerated>:1 +0x108
  github.com/cosmos/relayer/v2/cmd.(*ConfigInputWrapper).RuntimeConfig()
      /home/runner/work/relayer/relayer/cmd/config.go:337 +0x305
  github.com/cosmos/relayer/v2/cmd.(*appState).loadConfigFile()
      /home/runner/work/relayer/relayer/cmd/appstate.go:84 +0x391
  github.com/cosmos/relayer/v2/cmd.(*appState).performConfigLockingOperation()
      /home/runner/work/relayer/relayer/cmd/appstate.go:216 +0x324
  github.com/cosmos/relayer/v2/cmd.pathsNewCmd.func1()
      /home/runner/work/relayer/relayer/cmd/paths.go:246 +0x18d
  github.com/spf13/cobra.(*Command).execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0xf2f
  github.com/spf13/cobra.(*Command).ExecuteC()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x604
  github.com/spf13/cobra.(*Command).Execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039 +0x324
  github.com/spf13/cobra.(*Command).ExecuteContext()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1032 +0x31c
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunWithInputC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:83 +0x30e
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:57 +0x304
  github.com/cosmos/relayer/v2/interchaintest.(*Relayer).GeneratePath()
      /home/runner/work/relayer/relayer/interchaintest/relayer.go:114 +0x35
  github.com/strangelove-ventures/interchaintest/v8.(*Interchain).Build()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected][592](https://github.com/cosmos/relayer/actions/runs/6870999378/job/18687001943?pr=1334#step:5:593)933b/interchain.go:281 +0xd84
  github.com/strangelove-ventures/interchaintest/v8.StartChainPair()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/test_setup.go:73 +0x817
  github.com/strangelove-ventures/interchaintest/v8/conformance.TestChainPair()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:331 +0x891
  github.com/strangelove-ventures/interchaintest/v8/conformance.Test.func1.1.1.2()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:257 +0x259
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1[595](https://github.com/cosmos/relayer/actions/runs/6870999378/job/18687001943?pr=1334#step:5:596) +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44

Goroutine 1028 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x124
  github.com/cosmos/relayer/v2/relayer.(*Chain).CreateClients()
      /home/runner/work/relayer/relayer/relayer/client.go:73 +0xcbb
  github.com/cosmos/relayer/v2/cmd.createClientsCmd.func1()
      /home/runner/work/relayer/relayer/cmd/tx.go:102 +0x61c
  github.com/spf13/cobra.(*Command).execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0xf2f
  github.com/spf13/cobra.(*Command).ExecuteC()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x604
  github.com/spf13/cobra.(*Command).Execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039 +0x324
  github.com/spf13/cobra.(*Command).ExecuteContext()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1032 +0x31c
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunWithInputC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:83 +0x30e
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:57 +0x2f1
  github.com/cosmos/relayer/v2/interchaintest.(*Relayer).CreateClients()
      /home/runner/work/relayer/relayer/interchaintest/relayer.go:240 +0x2d
  github.com/strangelove-ventures/interchaintest/v8/conformance.TestRelayerSetup.func2()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/relayersetup.go:82 +0x20a
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44

Goroutine 23 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1[648](https://github.com/cosmos/relayer/actions/runs/6870999378/job/18687001943?pr=1334#step:5:649) +0x82a
  github.com/strangelove-ventures/interchaintest/v8/conformance.Test.func1.1.1()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:247 +0x31d
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44
==================
    testing.go:1465: race detected during execution of test

jtieri avatar Nov 15 '23 00:11 jtieri

We should look into if this is still needed for our feegrant test:

https://github.com/cosmos/relayer/blob/7e0276009d57837b7b1c8082c565b4f540db179e/relayer/chains/cosmos/bech32_hack.go#L16-L23

It seems like this has something to do with it.

boojamya avatar Nov 15 '23 00:11 boojamya

Good call out, i just dumped the logs and didn't really look into it much since it was end of my day. This method should really not be getting called in many places, it's the source of much pain and suffering. I'm surprised to see that it is used in any of our tests.

Once this is eventually closed out we will be able to kill this once and for all

jtieri avatar Nov 15 '23 02:11 jtieri

switch to docker image for e2e tests and use -race during compile in Dockerfile and then GORACE=halt_on_error=1 at runtime(https://github.com/strangelove-ventures/heighliner/issues/103)

akc2267 avatar Nov 20 '23 20:11 akc2267

Hey @KyleMoser now that this PR in the SDK has been merged, can we remove this code from the test case https://github.com/cosmos/relayer/issues/1335#issuecomment-1811617734?

It would be nice to fix the race conditions we see from time to time in the e2e test cases.

jtieri avatar Sep 12 '24 03:09 jtieri