libp2p
Detect failures related to TCP simultaneous open and retry
While working on my personal project, I noticed that when using the noise protocol, both peers are intermittently getting error messages when they try to connect to each other at the same time.
Link's project is a reproduction of that part. and I solved the problem by storing each other in the peer store and connecting streams without trying to connect directly at the moment.
The file below is a log.txt file included in the link, which shows that noise errors occur intermittently when a docker is used to create an isolated environment and try to connect by repeatedly creating/removing another peer with one peer.
root@.../go-noise-problem# docker run -it test
2024/08/17 06:07:48 Self: [/ip4/127.0.0.1/tcp/39859 /ip4/172.17.0.2/tcp/39859] 12D3KooWEy9zuUv1qc6JfswHGgvZxCVw2RWQwmzMMgwQC6xQz2RT
2024/08/17 06:08:02 Found: {12D3KooWAn1oVgmzmYF64xQpPL52rVVWatCQw5s8WDbToaXTxvYw: [/ip4/127.0.0.1/tcp/41965 /ip4/172.17.0.3/tcp/41965]}
2024/08/17 06:08:09 Found: {12D3KooW9ySZUUY7b7xhEUkPny5NMGaMeLvuyQmmF7U6ieznJo7Y: [/ip4/127.0.0.1/tcp/33779 /ip4/172.17.0.3/tcp/33779]}
2024/08/17 06:08:24 Found: {12D3KooWNkiPAMLLorVuqjAPWjo6w6WE5CFT67b1dWVWyg5Ruh4g: [/ip4/127.0.0.1/tcp/37107 /ip4/172.17.0.3/tcp/37107]}
2024/08/17 06:08:27 Found: {12D3KooWHSbowE11HW22qdD8EGWMdBwD8cNXX334bWxYdNWk2n5g: [/ip4/127.0.0.1/tcp/39443 /ip4/172.17.0.3/tcp/39443]}
2024/08/17 06:08:29 Found: {12D3KooWAT1YDAZm5ux9ZTrjDCrGM8dGXLxYZcfZszsXGBx9xRRG: [/ip4/127.0.0.1/tcp/39481 /ip4/172.17.0.3/tcp/39481]}
2024/08/17 06:08:31 Found: {12D3KooWML2GZt9nP4JPprdno4H2EgcWcxmMh9gc1MvrfNdmF6AA: [/ip4/127.0.0.1/tcp/35611 /ip4/172.17.0.3/tcp/35611]}
2024/08/17 06:08:32 Found: {12D3KooWJZr4WhUoyNLRzaCaJSrbZ79GTEqGPNKhcwdcG2SqPpRd: [/ip4/127.0.0.1/tcp/37463 /ip4/172.17.0.3/tcp/37463]}
2024/08/17 06:08:34 failed to dial: failed to dial 12D3KooWGdPNKAWySFjmWCJXPNP4UBcdtE9uT9eGJd3HyvPChAgD: all dials failed
* [/ip4/127.0.0.1/tcp/45231] dial tcp4 127.0.0.1:45231: connect: connection refused
* [/ip4/172.17.0.3/tcp/45231] failed to negotiate security protocol: error reading handshake message: noise: message is too short
If you want to reproduce the situation, you can reproduce it if you follow the sequence of test.sh .
thanks for reading and efforting.
Version Information
noise-test cloud.google.com/go v0.37.0 dmitri.shuralyov.com/app/changes v0.0.0-20180602232624-0a106ad413e3 dmitri.shuralyov.com/html/belt v0.0.0-20180602232347-f7d459c86be0 dmitri.shuralyov.com/service/change v0.0.0-20181023043359-a85b471d5412 dmitri.shuralyov.com/state v0.0.0-20180228185332-28bcc343414c git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999 git.sr.ht/~sbinet/gg v0.5.0 github.com/AllenDang/gform v0.0.0-20130830051017-6bb9f0d6b77e github.com/AllenDang/w32 v0.0.0-20180428130237-ad0a36d80adc github.com/AndreasBriese/bbloom v0.0.0-20190825152654-46b345b51c96 github.com/BurntSushi/toml v0.3.1 github.com/Jorropo/jsync v1.0.1 github.com/aead/siphash v1.0.1 github.com/ajstarks/svgo v0.0.0-20211024235047-1546f124cd8b github.com/alecthomas/kingpin/v2 v2.4.0 github.com/alecthomas/units v0.0.0-20231202071711-9a357b53e9c9 github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239 github.com/benbjohnson/clock v1.3.5 github.com/beorn7/perks v1.0.1 github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625 github.com/btcsuite/btcd v0.22.1 github.com/btcsuite/btcd/btcec/v2 v2.1.3 github.com/btcsuite/btcd/chaincfg/chainhash v1.0.1 github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f github.com/btcsuite/btcutil v0.0.0-20190207003914-4c204d697803 github.com/btcsuite/go-socks v0.0.0-20170105172521-4720035b7bfd github.com/btcsuite/goleveldb v0.0.0-20160330041536-7834afc9e8cd github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723 github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792 github.com/btcsuite/winsvc v1.0.0 github.com/buger/jsonparser v0.0.0-20181115193947-bf1c66bbce23 github.com/campoy/embedmd v1.0.0 github.com/cenkalti/backoff/v4 v4.3.0 github.com/census-instrumentation/opencensus-proto v0.2.1 github.com/cespare/xxhash v1.1.0 github.com/cespare/xxhash/v2 v2.3.0 github.com/chzyer/readline v1.5.1 github.com/cilium/ebpf v0.9.1 github.com/client9/misspell v0.3.4 github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f github.com/containerd/cgroups v1.1.0 github.com/coreos/go-semver v0.3.0 github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d github.com/coreos/go-systemd/v22 v22.5.0 github.com/cpuguy83/go-md2man/v2 v2.0.0 github.com/crackcomm/go-gitignore v0.0.0-20231225121904-e25f5bc08668 github.com/cskr/pubsub v1.0.2 github.com/davecgh/go-spew v1.1.1 github.com/davidlazar/go-crypto v0.0.0-20200604182044-b73af7476f6c github.com/decred/dcrd/crypto/blake256 v1.0.1 github.com/decred/dcrd/dcrec/secp256k1/v4 v4.3.0 github.com/dgraph-io/badger v1.6.2 github.com/dgraph-io/ristretto v0.0.2 github.com/docker/go-units v0.5.0 github.com/dustin/go-humanize v1.0.1 github.com/elastic/gosigar v0.14.3 github.com/envoyproxy/go-control-plane v0.9.4 github.com/envoyproxy/protoc-gen-validate v0.1.0 github.com/felixge/httpsnoop v1.0.4 github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568 github.com/flynn/noise v1.1.0 github.com/francoispqt/gojay v1.2.13 github.com/frankban/quicktest v1.14.6 github.com/fsnotify/fsnotify v1.4.7 github.com/gabriel-vasile/mimetype v1.4.4 github.com/ghodss/yaml v1.0.0 github.com/gliderlabs/ssh v0.1.1 github.com/go-errors/errors v1.0.1 github.com/go-fonts/liberation v0.3.2 github.com/go-kit/log v0.2.1 github.com/go-latex/latex v0.0.0-20231108140139-5c1ce85aa4ea github.com/go-logfmt/logfmt v0.5.1 github.com/go-logr/logr v1.4.2 github.com/go-logr/stdr v1.2.2 github.com/go-pdf/fpdf v0.9.0 github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 github.com/go-task/slim-sprig/v3 v3.0.0 github.com/go-yaml/yaml v2.1.0+incompatible github.com/goccmack/gocc v0.0.0-20230228185258-2292f9e40198 github.com/godbus/dbus/v5 v5.1.0 github.com/gogo/protobuf v1.3.2 github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da github.com/golang/lint v0.0.0-20180702182130-06c8688daad7 github.com/golang/mock v1.6.0 github.com/golang/protobuf v1.5.3 github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c github.com/google/go-cmp v0.6.0 github.com/google/go-github v17.0.0+incompatible github.com/google/go-querystring v1.0.0 github.com/google/gopacket v1.1.19 github.com/google/martian v2.1.0+incompatible github.com/google/pprof v0.0.0-20240727154555-813a5fbdbec8 github.com/google/renameio v0.1.0 github.com/google/uuid v1.6.0 github.com/googleapis/gax-go v2.0.0+incompatible github.com/googleapis/gax-go/v2 v2.0.3 github.com/gopherjs/gopherjs v0.0.0-20190430165422-3e4dfb77656c github.com/gorilla/mux v1.8.1 github.com/gorilla/websocket v1.5.3 github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 github.com/grpc-ecosystem/grpc-gateway v1.5.0 github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0 github.com/gxed/hashland/keccakpg v0.0.1 github.com/gxed/hashland/murmur3 v0.0.1 github.com/hashicorp/errwrap v1.1.0 github.com/hashicorp/go-multierror v1.1.1 github.com/hashicorp/golang-lru v1.0.2 github.com/hashicorp/golang-lru/arc/v2 v2.0.7 github.com/hashicorp/golang-lru/v2 v2.0.7 github.com/hpcloud/tail v1.0.0 github.com/huin/goupnp v1.3.0 github.com/ianlancetaylor/demangle v0.0.0-20240312041847-bd984b5ce465 github.com/ipfs/bbloom v0.0.4 github.com/ipfs/boxo v0.21.0 github.com/ipfs/go-bitfield v1.1.0 github.com/ipfs/go-block-format v0.2.0 github.com/ipfs/go-blockservice v0.5.2 github.com/ipfs/go-cid v0.4.1 github.com/ipfs/go-cidutil v0.1.0 github.com/ipfs/go-datastore v0.6.0 github.com/ipfs/go-detect-race v0.0.1 github.com/ipfs/go-ds-badger v0.3.0 github.com/ipfs/go-ds-leveldb v0.5.0 github.com/ipfs/go-ipfs-blockstore v1.3.1 github.com/ipfs/go-ipfs-blocksutil v0.0.1 github.com/ipfs/go-ipfs-delay v0.0.1 github.com/ipfs/go-ipfs-ds-help v1.1.1 github.com/ipfs/go-ipfs-exchange-interface v0.2.1 github.com/ipfs/go-ipfs-pq v0.0.3 github.com/ipfs/go-ipfs-redirects-file v0.1.1 github.com/ipfs/go-ipfs-util v0.0.3 github.com/ipfs/go-ipld-cbor v0.1.0 github.com/ipfs/go-ipld-format v0.6.0 github.com/ipfs/go-ipld-legacy v0.2.1 github.com/ipfs/go-log v1.0.5 github.com/ipfs/go-log/v2 v2.5.1 github.com/ipfs/go-merkledag v0.11.0 github.com/ipfs/go-metrics-interface v0.0.1 github.com/ipfs/go-peertaskqueue v0.8.1 github.com/ipfs/go-unixfs v0.4.5 github.com/ipfs/go-unixfsnode v1.9.0 github.com/ipfs/go-verifcid v0.0.3 github.com/ipld/go-car v0.6.2 github.com/ipld/go-car/v2 v2.13.1 github.com/ipld/go-codec-dagpb v1.6.0 github.com/ipld/go-ipld-prime v0.21.0 github.com/jackpal/go-nat-pmp v1.0.2 github.com/jbenet/go-cienv v0.1.0 github.com/jbenet/go-temp-err-catcher v0.1.0 github.com/jbenet/goprocess v0.1.4 github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1 github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89 github.com/jpillora/backoff v1.0.0 github.com/jrick/logrotate v1.0.0 github.com/json-iterator/go v1.1.12 github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024 github.com/jtolds/gls v4.20.0+incompatible github.com/julienschmidt/httprouter v1.3.0 github.com/kisielk/errcheck v1.5.0 github.com/kisielk/gotool v1.0.0 github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23 github.com/klauspost/compress v1.17.9 github.com/klauspost/cpuid/v2 v2.2.8 github.com/koron/go-ssdp v0.0.4 github.com/kr/pretty v0.3.1 github.com/kr/pty v1.1.3 github.com/kr/text v0.2.0 github.com/libp2p/go-buffer-pool v0.1.0 github.com/libp2p/go-cidranger v1.1.0 github.com/libp2p/go-doh-resolver v0.4.0 github.com/libp2p/go-flow-metrics v0.1.0 github.com/libp2p/go-libp2p v0.36.1 github.com/libp2p/go-libp2p-asn-util v0.4.1 github.com/libp2p/go-libp2p-core v0.20.1 github.com/libp2p/go-libp2p-crypto v0.1.0 github.com/libp2p/go-libp2p-kad-dht v0.25.2 github.com/libp2p/go-libp2p-kbucket v0.6.3 github.com/libp2p/go-libp2p-noise v0.5.0 github.com/libp2p/go-libp2p-peerstore v0.8.0 github.com/libp2p/go-libp2p-pubsub v0.11.0 github.com/libp2p/go-libp2p-record v0.2.0 github.com/libp2p/go-libp2p-routing-helpers v0.7.4 github.com/libp2p/go-libp2p-testing v0.12.0 github.com/libp2p/go-libp2p-xor v0.1.0 github.com/libp2p/go-libp2p-yamux v0.10.0 github.com/libp2p/go-msgio v0.3.0 github.com/libp2p/go-nat v0.2.0 github.com/libp2p/go-netroute v0.2.1 github.com/libp2p/go-openssl v0.1.0 github.com/libp2p/go-reuseport v0.4.0 github.com/libp2p/go-yamux/v3 v3.1.1 github.com/libp2p/go-yamux/v4 v4.0.1 github.com/libp2p/zeroconf/v2 v2.2.0 github.com/lunixbochs/vtclean v1.0.0 github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd github.com/mattn/go-isatty v0.0.20 github.com/mattn/go-pointer v0.0.1 github.com/matttproud/golang_protobuf_extensions v1.0.4 github.com/microcosm-cc/bluemonday v1.0.1 github.com/miekg/dns v1.1.61 github.com/mikioh/tcp v0.0.0-20190314235350-803a9b46060c github.com/mikioh/tcpinfo v0.0.0-20190314235526-30a79bb1804b github.com/mikioh/tcpopt v0.0.0-20190314235656-172688c1accc github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 github.com/minio/sha256-simd v1.0.1 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd github.com/modern-go/reflect2 v1.0.2 github.com/mr-tron/base58 v1.2.0 github.com/multiformats/go-base32 v0.1.0 github.com/multiformats/go-base36 v0.2.0 github.com/multiformats/go-multiaddr v0.13.0 github.com/multiformats/go-multiaddr-dns v0.3.1 github.com/multiformats/go-multiaddr-fmt v0.1.0 github.com/multiformats/go-multibase v0.2.0 github.com/multiformats/go-multicodec v0.9.0 github.com/multiformats/go-multihash v0.2.3 github.com/multiformats/go-multistream v0.5.0 github.com/multiformats/go-varint v0.0.7 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86 github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab github.com/nxadm/tail v1.4.11 github.com/onsi/ginkgo v1.16.5 github.com/onsi/ginkgo/v2 v2.19.1 github.com/onsi/gomega v1.34.0 github.com/opencontainers/runtime-spec v1.2.0 github.com/opentracing/opentracing-go v1.2.0 github.com/openzipkin/zipkin-go v0.4.3 github.com/pbnjay/memory v0.0.0-20210728143218-7b4eea64cf58 github.com/petar/GoLLRB v0.0.0-20210522233825-ae3b015fd3e9 github.com/pion/datachannel v1.5.8 github.com/pion/dtls/v2 v2.2.12 github.com/pion/ice/v2 v2.3.32 github.com/pion/interceptor v0.1.29 github.com/pion/logging v0.2.2 github.com/pion/mdns v0.0.12 github.com/pion/randutil v0.1.0 github.com/pion/rtcp v1.2.14 github.com/pion/rtp v1.8.8 github.com/pion/sctp v1.8.20 github.com/pion/sdp/v3 v3.0.9 github.com/pion/srtp/v2 v2.0.20 github.com/pion/stun v0.6.1 github.com/pion/transport/v2 v2.2.9 github.com/pion/transport/v3 v3.0.6 github.com/pion/turn/v2 v2.1.6 github.com/pion/webrtc/v3 v3.2.50 github.com/pkg/errors v0.9.1 github.com/pmezard/go-difflib v1.0.0 github.com/polydawn/refmt v0.89.0 github.com/prometheus/client_golang v1.19.1 github.com/prometheus/client_model v0.6.1 github.com/prometheus/common v0.55.0 github.com/prometheus/procfs v0.15.1 github.com/quic-go/qpack v0.4.0 github.com/quic-go/qtls-go1-20 v0.3.2 github.com/quic-go/quic-go v0.45.2 github.com/quic-go/webtransport-go v0.8.0 github.com/raulk/go-watchdog v1.3.0 github.com/rodrigocfd/windigo v0.0.0-20230809154420-8faa606d9f5f github.com/rogpeppe/go-internal v1.10.0 github.com/russross/blackfriday v1.5.2 github.com/russross/blackfriday/v2 v2.0.1 github.com/samber/lo v1.39.0 github.com/sclevine/agouti v3.0.0+incompatible github.com/sergi/go-diff v1.0.0 github.com/shurcooL/component v0.0.0-20170202220835-f88ec8f54cc4 github.com/shurcooL/events v0.0.0-20181021180414-410e4ca65f48 github.com/shurcooL/github_flavored_markdown v0.0.0-20181002035957-2122de532470 github.com/shurcooL/go v0.0.0-20180423040247-9e1955d9fb6e github.com/shurcooL/go-goon v0.0.0-20170922171312-37c2f522c041 github.com/shurcooL/gofontwoff v0.0.0-20180329035133-29b52fc0a18d github.com/shurcooL/gopherjslib v0.0.0-20160914041154-feb6d3990c2c github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b github.com/shurcooL/highlight_go v0.0.0-20181028180052-98c3abbbae20 github.com/shurcooL/home v0.0.0-20181020052607-80b7ffcb30f9 github.com/shurcooL/htmlg v0.0.0-20170918183704-d01228ac9e50 github.com/shurcooL/httperror v0.0.0-20170206035902-86b7830d14cc github.com/shurcooL/httpfs v0.0.0-20171119174359-809beceb2371 github.com/shurcooL/httpgzip v0.0.0-20180522190206-b1c53ac65af9 github.com/shurcooL/issues v0.0.0-20181008053335-6292fdc1e191 github.com/shurcooL/issuesapp v0.0.0-20180602232740-048589ce2241 github.com/shurcooL/notifications v0.0.0-20181007000457-627ab5aea122 github.com/shurcooL/octicon v0.0.0-20181028054416-fa4f57f9efb2 github.com/shurcooL/reactions v0.0.0-20181006231557-f2e0b4ca5b82 github.com/shurcooL/sanitized_anchor_name v1.0.0 github.com/shurcooL/users v0.0.0-20180125191416-49c67e49c537 github.com/shurcooL/webdavfs v0.0.0-20170829043945-18c3829fa133 github.com/sirupsen/logrus v1.8.1 github.com/smartystreets/assertions v1.2.0 github.com/smartystreets/goconvey v1.7.2 github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 github.com/spaolacci/murmur3 v1.1.0 github.com/stretchr/objx v0.5.2 github.com/stretchr/testify v1.9.0 github.com/syndtr/goleveldb v1.0.0 github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07 github.com/ucarion/urlpath v0.0.0-20200424170820-7ccc79b76bbb github.com/urfave/cli v1.22.10 github.com/viant/assertly v0.4.8 github.com/viant/toolbox v0.24.0 github.com/warpfork/go-testmark v0.12.1 github.com/warpfork/go-wish v0.0.0-20220906213052-39a1cc7a02d0 github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc github.com/whyrusleeping/cbor v0.0.0-20171005072247-63513f603b11 github.com/whyrusleeping/cbor-gen v0.1.2 github.com/whyrusleeping/chunker v0.0.0-20181014151217-fe64bd25879f github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1 github.com/wlynxg/anet v0.0.3 github.com/xhit/go-str2duration/v2 v2.1.0 github.com/yuin/goldmark v1.4.13 go.opencensus.io v0.24.0 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.52.0 go.opentelemetry.io/otel v1.28.0 go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.27.0 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.27.0 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.27.0 go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.27.0 go.opentelemetry.io/otel/exporters/zipkin v1.27.0 go.opentelemetry.io/otel/metric v1.28.0 go.opentelemetry.io/otel/sdk v1.27.0 go.opentelemetry.io/otel/trace v1.28.0 go.opentelemetry.io/proto/otlp v1.3.1 go.uber.org/atomic v1.11.0 go.uber.org/dig v1.17.1 go.uber.org/fx v1.22.1 go.uber.org/goleak v1.3.0 go.uber.org/mock v0.4.0 go.uber.org/multierr v1.11.0 go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee go.uber.org/zap v1.27.0 go4.org v0.0.0-20180809161055-417644f6feb5 golang.org/x/build v0.0.0-20190111050920-041ab4dc3f9d golang.org/x/crypto v0.25.0 golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 golang.org/x/image v0.14.0 golang.org/x/lint v0.0.0-20200302205851-738671d3881b golang.org/x/mod v0.19.0 golang.org/x/net v0.27.0 golang.org/x/oauth2 v0.21.0 golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852 golang.org/x/sync v0.7.0 golang.org/x/sys v0.22.0 golang.org/x/telemetry v0.0.0-20240521205824-bda55230c457 golang.org/x/term v0.22.0 golang.org/x/text v0.16.0 golang.org/x/time v0.5.0 golang.org/x/tools v0.23.0 golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 gonum.org/v1/gonum v0.15.0 gonum.org/v1/plot v0.14.0 google.golang.org/api v0.1.0 google.golang.org/appengine v1.6.7 google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013 google.golang.org/genproto/googleapis/api v0.0.0-20240617180043-68d350f18fd4 google.golang.org/genproto/googleapis/rpc v0.0.0-20240617180043-68d350f18fd4 google.golang.org/grpc v1.64.0 google.golang.org/protobuf v1.34.2 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c gopkg.in/errgo.v2 v2.1.0 gopkg.in/fsnotify.v1 v1.4.7 gopkg.in/inf.v0 v0.9.1 gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 gopkg.in/yaml.v2 v2.4.0 gopkg.in/yaml.v3 v3.0.1 grpc.go4.org v0.0.0-20170609214715-11d0a25b4919 honnef.co/go/tools v0.0.1-2019.2.3 lukechampine.com/blake3 v1.3.0 rsc.io/pdf v0.1.1 sourcegraph.com/sourcegraph/go-diff v0.5.0 sourcegraph.com/sqs/pbtypes v0.0.0-20180604144634-d3ebe8f20ae4
Thanks for the detailed report @elecbug! Have you noticed this issue in real networks, or just localhost/low-latency home networks?
My first suspicion is that this is related to TCP simultaneous open feature, since that causes confusion as to who is the server or client. Could you try running your repro with a small change for me to disprove this theory?
Please add these lines before your host.Connect
order := (bytes.Compare([]byte(host.ID()), []byte(found)))
if order == -1 {
time.Sleep(100*time.Millisecond)
}
// existing code below
err = host.Connect(ctx, found)
This basically has one side wait a bit if its peer id is lexicographically less. This should remove any chance for simultaneous open.
If the issue still happens after that change, please report back and I'll dive deeper.
The experimental environment was a dockers warm network built with about eight servers.
First of all, it seems that the error no longer occurs in the way you suggested. host.Connect trying at the same time seems to have been a problem.
One request, then can I know the code area where the near-simultaneous call of host.Connect between the two peers is causing the problem?
One request, then can I know the code area where the near-simultaneous call of
host.Connectbetween the two peers is causing the problem?
The "problem" is that TCP allows two endpoints to establish a connection if "two TCP simultaneously initiate the procedure" (https://datatracker.ietf.org/doc/html/rfc793#section-3.4). As far as I know, there's no way to detect if you're in this state or not using just Go's runtime + stdlib.
May I suggest using the QUIC stack instead? You won't have these issues there.
Closing this as it's only an issue in local environments, and there isn't a great fix.
Closing this as it's only an issue in local environments, and there isn't a great fix.
Note: This was fixed in multistream (not the best fix, but the best way to do it without breaking things), but then the fix was removed.
I had the same problem in my project, basically this occurs because you are using mDNS, when after you create the peer if you wait a while before starting mDNS this does not happen.
Really, the main issue is reuseport. An alternative is to use a different source port for outbound connections than the inbound port, but that makes it more difficult to detect port mappings in NATs, IIRC.
That's correct. reusing the port for dialing out lets us learn about our port mapping and our observed address as seen by our peer.
Maybe we should change the circumstances that trigger this. For example, in mDNS we can randomly delay publishing ourselves to avoid the simultaneous open in local environments.
It's not just MDNS. Any scenario causing two nodes to try to connect at the same time will trigger it, especially if those nodes are far apart (e.g., synchronized reconnect timers, etc.).
Sorry for not being clear. I did not say this is a problem in just mDNS. I outlined a way to avoid this problem when peers learn about each other at the same time via mDNS.
I'm not sure how common this over the internet. I've not found any data to suggest this happens frequently. And we have some data that shows we don't see this (https://github.com/libp2p/specs/issues/573). My intuition is that this is mainly a problem in networks with low latency such as the loopback interface and local networks, as peers can learn about each other quickly in those environments and send SYN packets to each other quickly.
If we accept the assumption that this is only an issue in local networks, we could not reuse the listen port for those networks. However, even if we do that, it would still be better to avoid having two connections when one would suffice. Therefore, changes to protocols that cause nodes to simultaneously connect would still be preferred.
You can detect simultaneous open in the security protocol, for example when a TLS client receives a TLS ClientHello. crypto/tls emits a really specific error. Then you can do a randomized backoff. Given how rare simultaneous open is in the wild, this doesn't cost you a lot.
Or just use QUIC, and avoid the entire simultaneous open nonsense...
You can detect simultaneous open in the security protocol, for example when a TLS client receives a TLS ClientHello. crypto/tls emits a really specific error. Then you can do a randomized backoff. Given how rare simultaneous open is in the wild, this doesn't cost you a lot.
Or just use QUIC, and avoid the entire simultaneous open nonsense...
Is it possible to implement quic-go with pnet? I would really like to know about this.
No, there's no way to do that, see https://github.com/libp2p/go-libp2p/issues/1432.
My intuition is that this is mainly a problem in networks with low latency such as the loopback interface and local networks, as peers can learn about each other quickly in those environments and send SYN packets to each other quickly.
It is more common in networks with high latency because SYN packets are more likely to cross on the wire.
Of course, you need a simultaneous trigger. But that can happen if an application has non-randomized reconnect timers and/or some central server signals two nodes to connect to each-other at the same time.
You can detect simultaneous open in the security protocol, for example when a TLS client receives a TLS ClientHello.
Yeah, this is the right way to fix it (that or use QUIC). And now that we have the holepunching logic, it should "just work" (return a retry error, backoff some amount of time (possibly even based on the peer IDs) and short-circuit if we see a new incoming connection).
Is it possible to implement quic-go with pnet?
@cpeliciari I'd love to hear why you use PNET here: https://github.com/libp2p/go-libp2p/discussions/2942.
You can detect simultaneous open in the security protocol...
I like this idea. Let's do it. We also have to handle the case for when the failure happens in the multistream select layer, but it should be straightforward.
Reopening to track this fix.