p2plab
p2plab copied to clipboard
Published
20 hours ago •
Netflix
Netflix
Try out go-libp2p-noise@begin-noise
Trying out a new secure transport from ChainSafe: https://github.com/ChainSafe/go-libp2p-noise/blob/master/go-libp2p-noise-ethberlin-1a.pdf
Test transferring OCI image over tcp + mplex + noise:
$ git clone [email protected]:Netflix/p2plab.git
$ cd p2plab
$ git checkout libp2p-noise
$ export GO111MODULE=on
$ go run ./cmd/ociadd docker.io/library/alpine:latest
7:01PM INF Starting libp2p peer id=QmdtV78HZThEvVLrGyW3uEZYTLj57TfPR6uVDHRra2KP8q listen=["/ip4/127.0.0.1/tcp/41159","/ip4/192.168.87.35/tcp/41159","/ip4/192.168.122.1/tcp/41159","/ip4/172.17.0.1/tcp/41159","/ip4/192.168.1.6/tcp/41159"]
7:01PM INF Converted OCI image to IPLD DAG cid=bafybeifbgj6mwrb5zg7yekykmeb73tpqowcd5tkevdnurbnmlxaxm7ha5a ref=docker.io/library/alpine:latest
7:01PM INF Retrieve manifest from another p2plab/peer by running:
go run ./cmd/ociget /ip4/127.0.0.1/tcp/41159/p2p/QmdtV78HZThEvVLrGyW3uEZYTLj57TfPR6uVDHRra2KP8q bafybeifbgj6mwrb5zg7yekykmeb73tpqowcd5tkevdnurbnmlxaxm7ha5a
7:01PM INF Connect to this peer from IPFS daemon:
ipfs swarm connect /ip4/127.0.0.1/tcp/41159/p2p/QmdtV78HZThEvVLrGyW3uEZYTLj57TfPR6uVDHRra2KP8q
ipfs cat bafybeifbgj6mwrb5zg7yekykmeb73tpqowcd5tkevdnurbnmlxaxm7ha5a
Press 'Enter' to terminate peer...
$ export GO111MODULE=on
$ go run ./cmd/ociget /ip4/127.0.0.1/tcp/41159/p2p/QmdtV78HZThEvVLrGyW3uEZYTLj57TfPR6uVDHRra2KP8q bafybeifbgj6mwrb5zg7yekykmeb73tpqowcd5tkevdnurbnmlxaxm7ha5a
7:01PM INF Starting libp2p peer id=QmUH3yZLjfLJNbKMXtPKrzNMYS1YG7bQDCeheRBkefduc1 listen=["/ip4/127.0.0.1/tcp/46731/ws","/ip4/192.168.87.35/tcp/46731/ws","/ip4/192.168.122.1/tcp/46731/ws","/ip4/172.17.0.1/tcp/46731/ws","/ip4/192.168.1.6/tcp/46731/ws","/ip4/127.0.0.1/udp/51444/quic","/ip4/192.168.87.35/udp/51444/quic","/ip4/192.168.122.1/udp/51444/quic","/ip4/172.17.0.1/udp/51444/quic","/ip4/192.168.1.6/udp/51444/quic","/ip4/127.0.0.1/tcp/33373","/ip4/192.168.87.35/tcp/33373","/ip4/192.168.122.1/tcp/33373","/ip4/172.17.0.1/tcp/33373","/ip4/192.168.1.6/tcp/33373"]
7:01PM INF Connected to peer addr=/ip4/127.0.0.1/tcp/41159/ipfs/QmdtV78HZThEvVLrGyW3uEZYTLj57TfPR6uVDHRra2KP8q
7:01PM INF Found file in unixfs directory name=sha256:71fe3723a9951fa7523d58253eb59d75a93269f2938a6229419de32e8f92b928
7:01PM INF Found file in unixfs directory name=sha256:778bd6b2922bd8c71c5ca7b5f4478c9e9bddb12ddf67ff0a575adb538a06d0e6
Press 'Enter' to terminate peer...
Test transferring docker.io/library/golang:latest from two neighbors peers using inmemory driver:
$ export GO111MODULE=on
$ go run ./cmd/labd
$ export GO111MODULE=on
$ go install ./cmd/labctl
$ labctl cluster create --definition examples/cluster/same-region.json same-region
6:54PM INF Creating node group name=same-region
6:54PM INF Updating metadata with new nodes name=same-region
6:54PM INF Waiting for healthy nodes name=same-region
6:54PM INF Updating cluster metadata name=same-region
6:54PM INF Created cluster "same-region"
same-region
$ labctl node update --ref libp2p-noise --security-transports noise same-region
+----------------------+-----------+--------------+---------------------------------------------------+----------------+-----------+
| ID | ADDRESS | GITREFERENCE | LABELS | CREATEDAT | UPDATEDAT |
+----------------------+-----------+--------------+---------------------------------------------------+----------------+-----------+
| blpvpv7ic6vf7a3g5in0 | 127.0.0.1 | libp2p-noise | blpvpv7ic6vf7a3g5in0,t2.micro,us-west-2 | 54 seconds ago | now |
| blpvpvfic6vf7a3g5ing | 127.0.0.1 | libp2p-noise | blpvpvfic6vf7a3g5ing,neighbors,t2.micro,us-west-2 | 54 seconds ago | now |
| blpvpvnic6vf7a3g5io0 | 127.0.0.1 | libp2p-noise | blpvpvnic6vf7a3g5io0,neighbors,t2.micro,us-west-2 | 54 seconds ago | now |
+----------------------+-----------+--------------+---------------------------------------------------+----------------+-----------+
$ labctl --output json node ls same-region | jq '.[0].Peer'
{
"GitReference": "libp2p-noise",
"Transports": [
"tcp"
],
"Muxers": [
"mplex"
],
"SecurityTransports": [
"noise"
],
"Routing": "nil"
}
$ labctl scenario create examples/scenario/neighbors.json
6:57PM INF Created scenario "neighbors"
neighbors
$ labctl benchmark create same-region neighbors
6:57PM INF Retrieving nodes in cluster bid=same-region-neighbors-1567882651553383197
6:57PM INF Resolving git references bid=same-region-neighbors-1567882651553383197
6:57PM INF Building p2p app(s) bid=same-region-neighbors-1567882651553383197 commits=["e13af9d394498f55acdb7951b122ca99ac9816f1"]
6:57PM INF Updating cluster bid=same-region-neighbors-1567882651553383197
6:57PM INF Retrieving peer infos bid=same-region-neighbors-1567882651553383197
6:57PM INF Connecting cluster bid=same-region-neighbors-1567882651553383197
6:57PM INF Creating scenario plan bid=same-region-neighbors-1567882651553383197
6:57PM INF Transforming objects into IPLD DAGs bid=same-region-neighbors-1567882651553383197
6:57PM INF Resolving OCI reference bid=same-region-neighbors-1567882651553383197 source=docker.io/library/golang:latest
6:57PM INF Resolved reference to digest bid=same-region-neighbors-1567882651553383197 digest=sha256:30101c5d45c7b68c2233962251da3d599417055e5ad73b84b7c4323b11caf57e source=docker.io/library/golang:latest
6:57PM INF Converting manifest recursively to IPLD DAG bid=same-region-neighbors-1567882651553383197 digest=sha256:30101c5d45c7b68c2233962251da3d599417055e5ad73b84b7c4323b11caf57e
6:58PM INF Constructing Unixfs directory over manifest blobs bid=same-region-neighbors-1567882651553383197 target=sha256:e7c6d1d3c099cc16fbee65106b15abd03ad54645702e3204054d4a27e986b31a
6:58PM INF Planning scenario seed bid=same-region-neighbors-1567882651553383197
6:58PM INF Planning scenario benchmark bid=same-region-neighbors-1567882651553383197
6:58PM INF Creating benchmark metadata bid=same-region-neighbors-1567882651553383197
6:58PM INF Executing scenario plan bid=same-region-neighbors-1567882651553383197
6:58PM INF Seeding cluster bid=same-region-neighbors-1567882651553383197
6:58PM INF Seeding cluster bid=same-region-neighbors-1567882651553383197 elapsed="20 seconds"
6:58PM INF Seeding cluster bid=same-region-neighbors-1567882651553383197 elapsed="40 seconds"
6:59PM INF Seeding cluster bid=same-region-neighbors-1567882651553383197 elapsed="1 minute"
// ...
Unfortunately, it doesn't seem to successfully complete atm.
For example, we can run the benchmark successfully using secio:
$ labctl node update --security-transports secio same-region
+----------------------+-----------+--------------+---------------------------------------------------+----------------+-----------+
| ID | ADDRESS | GITREFERENCE | LABELS | CREATEDAT | UPDATEDAT |
+----------------------+-----------+--------------+---------------------------------------------------+----------------+-----------+
| blpvpv7ic6vf7a3g5in0 | 127.0.0.1 | libp2p-noise | blpvpv7ic6vf7a3g5in0,t2.micro,us-west-2 | 15 minutes ago | now |
| blpvpvfic6vf7a3g5ing | 127.0.0.1 | libp2p-noise | blpvpvfic6vf7a3g5ing,neighbors,t2.micro,us-west-2 | 15 minutes ago | now |
| blpvpvnic6vf7a3g5io0 | 127.0.0.1 | libp2p-noise | blpvpvnic6vf7a3g5io0,neighbors,t2.micro,us-west-2 | 15 minutes ago | now |
+----------------------+-----------+--------------+---------------------------------------------------+----------------+-----------+
$ labctl benchmark create same-region neighbors
7:10PM INF Retrieving nodes in cluster bid=same-region-neighbors-1567883439546208960
7:10PM INF Resolving git references bid=same-region-neighbors-1567883439546208960
7:10PM INF Building p2p app(s) bid=same-region-neighbors-1567883439546208960 commits=["e13af9d394498f55acdb7951b122ca99ac9816f1"]
7:10PM INF Updating cluster bid=same-region-neighbors-1567883439546208960
7:10PM INF Retrieving peer infos bid=same-region-neighbors-1567883439546208960
7:10PM INF Connecting cluster bid=same-region-neighbors-1567883439546208960
7:10PM INF Creating scenario plan bid=same-region-neighbors-1567883439546208960
7:10PM INF Transforming objects into IPLD DAGs bid=same-region-neighbors-1567883439546208960
7:10PM INF Resolving OCI reference bid=same-region-neighbors-1567883439546208960 source=docker.io/library/golang:latest
7:10PM INF Resolved reference to digest bid=same-region-neighbors-1567883439546208960 digest=sha256:30101c5d45c7b68c2233962251da3d599417055e5ad73b84b7c4323b11caf57e source=docker.io/library/golang:latest
7:10PM INF Constructing Unixfs directory over manifest blobs bid=same-region-neighbors-1567883439546208960 target=sha256:e7c6d1d3c099cc16fbee65106b15abd03ad54645702e3204054d4a27e986b31a
7:10PM INF Planning scenario seed bid=same-region-neighbors-1567883439546208960
7:10PM INF Planning scenario benchmark bid=same-region-neighbors-1567883439546208960
7:10PM INF Creating benchmark metadata bid=same-region-neighbors-1567883439546208960
7:10PM INF Executing scenario plan bid=same-region-neighbors-1567883439546208960
7:10PM INF Seeding cluster bid=same-region-neighbors-1567883439546208960
7:10PM INF Seeding completed bid=same-region-neighbors-1567883439546208960
7:10PM INF Benchmarking cluster bid=same-region-neighbors-1567883439546208960
7:10PM INF Benchmark completed bid=same-region-neighbors-1567883439546208960
7:10PM INF Retrieving reports bid=same-region-neighbors-1567883439546208960
7:10PM INF Updating benchmark metadata bid=same-region-neighbors-1567883439546208960
7:10PM INF Completed benchmark "same-region-neighbors-1567883439546208960"
# Summary
Total time: 2 seconds 621 milliseconds
Trace: http://localhost:16686/trace/39fe431082ff12f2?uiFind=39fe431082ff12f2
# Bandwidth
+-------------------+----------------------+---------+----------+----------+----------+
| QUERY | NODE | TOTALIN | TOTALOUT | RATEIN | RATEOUT |
+-------------------+----------------------+---------+----------+----------+----------+
| (not 'neighbors') | blpvpv7ic6vf7a3g5in0 | 292 MB | 152 kB | 126 MB/s | 64 kB/s |
+-------------------+----------------------+---------+----------+----------+----------+
| - | blpvpvfic6vf7a3g5ing | 326 MB | 86 MB | 20 MB/s | 43 MB/s |
+ +----------------------+---------+----------+----------+----------+
| | blpvpvnic6vf7a3g5io0 | 329 MB | 255 MB | 19 MB/s | 91 MB/s |
+-------------------+----------------------+---------+----------+----------+----------+
| TOTAL | 947 MB | 342 MB | 165 MB/s | 133 MB/s |
+-------------------+----------------------+---------+----------+----------+----------+
# Bitswap
+-------------------+----------------------+------------+------------+-----------+----------+----------+---------+
| QUERY | NODE | BLOCKSRECV | BLOCKSSENT | DUPBLOCKS | DATARECV | DATASENT | DUPDATA |
+-------------------+----------------------+------------+------------+-----------+----------+----------+---------+
| (not 'neighbors') | blpvpv7ic6vf7a3g5in0 | 1,573 | 0 | 14 | 407 MB | 0 B | 3.7 MB |
+-------------------+----------------------+------------+------------+-----------+----------+----------+---------+
| - | blpvpvfic6vf7a3g5ing | 1,267 | 554 | 67 | 326 MB | 140 MB | 17 MB |
+ +----------------------+------------+------------+-----------+----------+----------+---------+
| | blpvpvnic6vf7a3g5io0 | 1,290 | 1,184 | 87 | 329 MB | 305 MB | 20 MB |
+-------------------+----------------------+------------+------------+-----------+----------+----------+---------+
| TOTAL | 4,130 | 1,738 | 168 | 1.1 GB | 445 MB | 41 MB |
+-------------------+----------------------+------------+------------+-----------+----------+----------+---------+
Hey, could you re-run the output with any extra debug logs turned on? Trying to understand what's happening in this code: https://github.com/Netflix/p2plab/blob/72fbd93500fa266a1790b6d427b814e9b20a6cdf/scenarios/run.go#L61 Wondering if there are any particular errors, or if it just hangs. Also, what is this function actually doing in terms of the libp2p nodes?
Sure, let me try to get the noise debug logs to you.
The benchmark consists of two stages, Seed and Benchmark. They pretty much do the same thing, except the latter is measured. Seeding prepares the scenario for the benchmark.
For example, in the neighbors scenario:
https://github.com/Netflix/p2plab/blob/72fbd93500fa266a1790b6d427b814e9b20a6cdf/examples/scenario/neighbors.json#L8-L13
We are seeding the nodes with the label neighbors with the golang IPLD DAG, then in the benchmark, measuring how long it takes for nodes without the label neighbors retrieves the golang DAG from its network. In this case, the network is the two other nodes with the label neighbors.
We are using bitswap as an exchange over the libp2p peer, and attempting to fetch the whole DAG by using a ipld.NodeGetter.
https://github.com/Netflix/p2plab/blob/72fbd93500fa266a1790b6d427b814e9b20a6cdf/dag/walker.go#L26-L39
@noot noise.log
Notable logs:
11:03:49.397 ERROR noise: write length err write tcp4 172.17.0.1:45249->192.168.87.35:34533: write: connection reset by peer protocol.go:302
11:03:49.398 DEBUG noise: verifyPayload msg=6e6f6973652d6c69627032702d7374617469632d6b65793acffae7a02ee780905ac1158534fd26adc54cca2f0740eb96c4d63912a60f2c6c protocol.go:139
11:03:49.398 DEBUG noise: runHandshake_xx stage=2 initiator=false remote key=cffae7a02ee780905ac1158534fd26adc54cca2f0740eb96c4d63912a60f2c6c xx_handshake.go:240
11:03:49.398 DEBUG noise: runHandshake_xx done initiator=false xx_handshake.go:275
11:03:49.399 DEBUG noise: verifyPayload msg=6e6f6973652d6c69627032702d7374617469632d6b65793acffae7a02ee780905ac1158534fd26adc54cca2f0740eb96c4d63912a60f2c6c protocol.go:139
11:03:49.399 ERROR noise: write length err write tcp4 172.17.0.1:45249->192.168.87.35:45725: write: connection reset by peer protocol.go:302
11:03:49.399 DEBUG noise: runHandshake_xx done initiator=false xx_handshake.go:275
11:03:49.399 DEBUG noise: runHandshake_xx done initiator=false xx_handshake.go:275
11:03:49.399 ERROR noise: write length err write tcp4 127.0.0.1:45249->127.0.0.1:45725: write: connection reset by peer protocol.go:302
11:03:49.399 DEBUG noise: runHandshake_xx stage=2 initiator=false remote key=e5c306c88cee9cfa927b777d005092f439713c1f8ced8024d05fbffe7287fb0d xx_handshake.go:240
11:03:49.399 ERROR noise: write length err write tcp4 192.168.122.1:45249->192.168.122.1:34533: write: connection reset by peer protocol.go:302
11:03:49.399 DEBUG noise: verifyPayload msg=6e6f6973652d6c69627032702d7374617469632d6b65793ae5c306c88cee9cfa927b777d005092f439713c1f8ced8024d05fbffe7287fb0d protocol.go:139
11:03:49.399 DEBUG noise: runHandshake_xx done initiator=false xx_handshake.go:275
11:03:49.399 ERROR noise: write length err write tcp4 192.168.87.35:45249->192.168.87.35:45725: write: connection reset by peer protocol.go:302
You can reproduce this on your machine by running labd with:
$ export GO111MODULE=on
$ export IPFS_LOGGING=DEBUG
$ export GOLOG_FILE=noise.log
$ go run ./cmd/labd
Hey, I'm having some issues running the code:
go: finding github.com/hinshun/go-bitswap v0.1.7-0.20190910164641-980f3a3f8fed
go: github.com/hinshun/[email protected]: unknown revision 980f3a3f8fed
go: github.com/hinshun/[email protected]: unknown revision 980f3a3f8fed
go: github.com/hinshun/[email protected]: unknown revision 980f3a3f8fed
go: error loading module requirements
Hi @noot, sorry for the trouble on the master branch. You should checkout libp2p-noise branch for now.
No worries, on the libp2p-noise branch I don't actually see any noise logs, I only see stuff for addrutil:
14:02:52.874 INFO badger: All 0 tables opened in 0s
logger.go:46
14:02:52.882 INFO badger: Replaying file id: 0 at offset: 0
logger.go:46
14:02:52.883 INFO badger: Replay took: 29.552µs
logger.go:46
14:02:53.068 DEBUG addrutil: InterfaceAddresses: from manet: [/ip4/127.0.0.1 /ip4/192.168.1.144 /ip4/172.17.0.1 /ip4/172.21.0.1 /ip6/::1 /ip6/fe80::197c:a4c1:16cb:8398] addr.go:121
14:02:53.068 DEBUG addrutil: InterfaceAddresses: usable: [/ip4/127.0.0.1 /ip4/192.168.1.144 /ip4/172.17.0.1 /ip4/172.21.0.1 /ip6/::1] addr.go:133
14:02:53.068 DEBUG addrutil: adding resolved addr: /ip4/0.0.0.0/tcp/42569 /ip4/127.0.0.1/tcp/42569 [/ip4/127.0.0.1/tcp/42569] addr.go:64
14:02:53.068 DEBUG addrutil: adding resolved addr: /ip4/0.0.0.0/tcp/42569 /ip4/192.168.1.144/tcp/42569 [/ip4/127.0.0.1/tcp/42569 /ip4/192.168.1.144/tcp/42569] addr.go:64
14:02:53.068 DEBUG addrutil: adding resolved addr: /ip4/0.0.0.0/tcp/42569 /ip4/172.17.0.1/tcp/42569 [/ip4/127.0.0.1/tcp/42569 /ip4/192.168.1.144/tcp/42569 /ip4/172.17.0.1/tcp/42569] addr.go:64
14:02:53.068 DEBUG addrutil: adding resolved addr: /ip4/0.0.0.0/tcp/42569 /ip4/172.21.0.1/tcp/42569 [/ip4/127.0.0.1/tcp/42569 /ip4/192.168.1.144/tcp/42569 /ip4/172.17.0.1/tcp/42569 /ip4/172.21.0.1/tcp/42569] addr.go:64
14:02:53.068 DEBUG addrutil: adding resolved addr: /ip4/0.0.0.0/tcp/33783/ws /ip4/127.0.0.1/tcp/33783/ws [/ip4/127.0.0.1/tcp/33783/ws] addr.go:64
14:02:53.068 DEBUG addrutil: adding resolved addr: /ip4/0.0.0.0/tcp/33783/ws /ip4/192.168.1.144/tcp/33783/ws [/ip4/127.0.0.1/tcp/33783/ws /ip4/192.168.1.144/tcp/33783/ws] addr.go:64
14:02:53.068 DEBUG addrutil: adding resolved addr: /ip4/0.0.0.0/tcp/33783/ws /ip4/172.17.0.1/tcp/33783/ws [/ip4/127.0.0.1/tcp/33783/ws /ip4/192.168.1.144/tcp/33783/ws /ip4/172.17.0.1/tcp/33783/ws] addr.go:64
14:02:53.068 DEBUG addrutil: adding resolved addr: /ip4/0.0.0.0/tcp/33783/ws /ip4/172.21.0.1/tcp/33783/ws [/ip4/127.0.0.1/tcp/33783/ws /ip4/192.168.1.144/tcp/33783/ws /ip4/172.17.0.1/tcp/33783/ws /ip4/172.21.0.1/tcp/33783/ws] addr.go:64
14:02:53.069 DEBUG addrutil: adding resolved addr: /ip4/0.0.0.0/udp/45981/quic /ip4/127.0.0.1/udp/45981/quic [/ip4/127.0.0.1/udp/45981/quic] addr.go:64
14:02:53.069 DEBUG addrutil: adding resolved addr: /ip4/0.0.0.0/udp/45981/quic /ip4/192.168.1.144/udp/45981/quic [/ip4/127.0.0.1/udp/45981/quic /ip4/192.168.1.144/udp/45981/quic] addr.go:64
14:02:53.069 DEBUG addrutil: adding resolved addr: /ip4/0.0.0.0/udp/45981/quic /ip4/172.17.0.1/udp/45981/quic [/ip4/127.0.0.1/udp/45981/quic /ip4/192.168.1.144/udp/45981/quic /ip4/172.17.0.1/udp/45981/quic] addr.go:64
14:02:53.069 DEBUG addrutil: adding resolved addr: /ip4/0.0.0.0/udp/45981/quic /ip4/172.21.0.1/udp/45981/quic [/ip4/127.0.0.1/udp/45981/quic /ip4/192.168.1.144/udp/45981/quic /ip4/172.17.0.1/udp/45981/quic /ip4/172.21.0.1/udp/45981/quic] addr.go:64
14:02:53.069 DEBUG addrutil: ResolveUnspecifiedAddresses: [/p2p-circuit /ip4/0.0.0.0/tcp/42569 /ip4/0.0.0.0/tcp/33783/ws /ip4/0.0.0.0/udp/45981/quic] [/ip4/127.0.0.1 /ip4/192.168.1.144 /ip4/172.17.0.1 /ip4/172.21.0.1 /ip6/::1] [/p2p-circuit /ip4/127.0.0.1/tcp/42569 /ip4/192.168.1.144/tcp/42569 /ip4/172.17.0.1/tcp/42569 /ip4/172.21.0.1/tcp/42569 /ip4/127.0.0.1/tcp/33783/ws /ip4/192.168.1.144/tcp/33783/ws /ip4/172.17.0.1/tcp/33783/ws /ip4/172.21.0.1/tcp/33783/ws /ip4/127.0.0.1/udp/45981/quic /ip4/192.168.1.144/udp/45981/quic /ip4/172.17.0.1/udp/45981/quic /ip4/172.21.0.1/udp/45981/quic] addr.go:109
It seems to hang after that, how long am I supposed to run it for?
But based on the logs you showed, it looks like the connection is being reset by the initiator of the connection, it doesn't look like the initiator is completing the handshake. There is probably some bug.
@noot Did you run the benchmark? Those logs indicate you only ran the labd daemon. I’ve included the full reproduction steps in the PR description.
@hinshun Hey, sorry for the late reply. Could you run it again after go get github.com/ChainSafe/go-libp2p-noise@master and let me know if there is any change? If not, I'll take another look.