linkerd-tcp
linkerd-tcp copied to clipboard
Published
20 hours ago •
linkerd
linkerd
linkerd-tcp connection reuse issue
trafficstars
Running a load test without connection reuse (this is part of my testing out the tacho OrderMap changes) and linkerd-tcp starts rejecting traffic after a thousand requests.
stevej@netty-test-8:~$ ./slow_cooker_linux_amd64 -noreuse -host 'default' -totalRequests 1000000 -qps 100 -concurrency 100 http://proxy-test-4d:7474/
# sending 10000 req/s with concurrency=100 to http://proxy-test-4d:7474/ ...
# good/b/f t good% min [p50 p95 p99 p999] max change
2017-04-20T21:47:03Z 14105/0/0 100000 14% 10s 5 [ 44 47 48 54 ] 54 +
2017-04-20T21:47:13Z 0/0/0 100000 0% 10s 0 [ 0 0 0 0 ] 0 -
2017-04-20T21:47:23Z 1039/0/0 100000 1% 10s 40 [ 54 15319 15327 15335 ] 15329 +++
2017-04-20T21:47:33Z 780/0/0 100000 0% 10s 40 [1060 3081 3083 3083 ] 3083
2017-04-20T21:47:43Z 659/0/0 100000 0% 10s 44 [1048 3073 3075 4073 ] 4073
CPU was at 1% and traffic was trickling through. Here's some example strace output
recvfrom(78, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(70, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(58, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(70, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(65, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(51, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(65, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(60, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(46, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(60, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(53, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(39, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(53, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(48, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(33, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(48, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(41, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(27, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(41, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(36, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(20, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(36, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(30, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(17, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(30, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(23, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(14, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(23, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
and an lsof report
stevej@proxy-test-4d:~$ sudo lsof -p 14178
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
linkerd-t 14178 stevej cwd DIR 8,1 4096 1812484 /home/stevej/src/linkerd-tcp
linkerd-t 14178 stevej rtd DIR 8,1 4096 2 /
linkerd-t 14178 stevej txt REG 8,1 9851408 1816829 /home/stevej/src/linkerd-tcp/target/release/
linkerd-tcp
linkerd-t 14178 stevej mem REG 8,1 1088952 1907 /lib/x86_64-linux-gnu/libm-2.24.so
linkerd-t 14178 stevej mem REG 8,1 1856752 1903 /lib/x86_64-linux-gnu/libc-2.24.so
linkerd-t 14178 stevej mem REG 8,1 92552 1931 /lib/x86_64-linux-gnu/libgcc_s.so.1
linkerd-t 14178 stevej mem REG 8,1 142400 1918 /lib/x86_64-linux-gnu/libpthread-2.24.so
linkerd-t 14178 stevej mem REG 8,1 31712 1920 /lib/x86_64-linux-gnu/librt-2.24.so
linkerd-t 14178 stevej mem REG 8,1 14608 1906 /lib/x86_64-linux-gnu/libdl-2.24.so
linkerd-t 14178 stevej mem REG 8,1 158512 1899 /lib/x86_64-linux-gnu/ld-2.24.so
linkerd-t 14178 stevej 0u CHR 136,3 0t0 6 /dev/pts/3
linkerd-t 14178 stevej 1u CHR 136,3 0t0 6 /dev/pts/3
linkerd-t 14178 stevej 2u CHR 136,3 0t0 6 /dev/pts/3
linkerd-t 14178 stevej 3u a_inode 0,11 0 8469 [eventpoll]
linkerd-t 14178 stevej 4r FIFO 0,10 0t0 908911732 pipe
linkerd-t 14178 stevej 5w FIFO 0,10 0t0 908911732 pipe
linkerd-t 14178 stevej 6u IPv4 908911733 0t0 TCP *:7474 (LISTEN)
linkerd-t 14178 stevej 7u a_inode 0,11 0 8469 [eventpoll]
linkerd-t 14178 stevej 8r FIFO 0,10 0t0 908910932 pipe
linkerd-t 14178 stevej 9w FIFO 0,10 0t0 908910932 pipe
linkerd-t 14178 stevej 10u IPv4 908911215 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33709 (ESTABLISHED)
linkerd-t 14178 stevej 11u IPv4 908911205 0t0 TCP proxy-test-4d:54924->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 12u IPv4 908910933 0t0 TCP *:9989 (LISTEN)
linkerd-t 14178 stevej 13u IPv4 908910934 0t0 TCP localhost:50620->localhost:4180 (ESTABLISHED
)
linkerd-t 14178 stevej 14u IPv4 908910257 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33616 (ESTABLISHED)
linkerd-t 14178 stevej 15u IPv4 908910258 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33604 (ESTABLISHED)
linkerd-t 14178 stevej 16u IPv4 908910259 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33609 (ESTABLISHED)
linkerd-t 14178 stevej 17u IPv4 908910260 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33622 (ESTABLISHED)
linkerd-t 14178 stevej 18u IPv4 908910261 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33624 (ESTABLISHED)
linkerd-t 14178 stevej 19u IPv4 908910262 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33620 (ESTABLISHED)
[cutting hundreds of other sockets that are open both upstream and downstream]
linkerd-t 14178 stevej 327u IPv4 908911328 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33773 (ESTABLISHED)
linkerd-t 14178 stevej 328u IPv4 908911331 0t0 TCP proxy-test-4d:42162->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 329u IPv4 908911332 0t0 TCP proxy-test-4d:7474->netty-test-8.c[124/1893]osted.internal:33770 (ESTABLISHED)
linkerd-t 14178 stevej 330u IPv4 908911333 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33772 (ESTABLISHED)
linkerd-t 14178 stevej 331u IPv4 908911334 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33775 (ESTABLISHED)
linkerd-t 14178 stevej 332u IPv4 908911335 0t0 TCP proxy-test-4d:38944->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 333u IPv4 908911336 0t0 TCP proxy-test-4d:55042->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 334u IPv4 908911337 0t0 TCP proxy-test-4d:42168->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 335u IPv4 908911338 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33769 (ESTABLISHED)
linkerd-t 14178 stevej 336u IPv4 908911339 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33771 (ESTABLISHED)
linkerd-t 14178 stevej 337u IPv4 908911340 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33765 (ESTABLISHED)
linkerd-t 14178 stevej 338u IPv4 908911343 0t0 TCP proxy-test-4d:38954->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 339u IPv4 908911344 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33777 (ESTABLISHED)
linkerd-t 14178 stevej 340u IPv4 908911345 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33778 (ESTABLISHED)
linkerd-t 14178 stevej 341u IPv4 908911346 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33776 (ESTABLISHED)
linkerd-t 14178 stevej 342u IPv4 908911347 0t0 TCP proxy-test-4d:38956->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 343u IPv4 908911348 0t0 TCP proxy-test-4d:55054->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 344u IPv4 908911349 0t0 TCP proxy-test-4d:42180->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 345u IPv4 908911350 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33780 (ESTABLISHED)
linkerd-t 14178 stevej 346u IPv4 908911351 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33782 (ESTABLISHED)
linkerd-t 14178 stevej 347u IPv4 908911352 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33781 (ESTABLISHED)
linkerd-t 14178 stevej 348u IPv4 908911353 0t0 TCP proxy-test-4d:42182->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 349u IPv4 908911354 0t0 TCP proxy-test-4d:55060->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 350u IPv4 908911355 0t0 TCP proxy-test-4d:38966->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 351u IPv4 908911356 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33779 (ESTABLISHED)
linkerd-t 14178 stevej 352u IPv4 908911357 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33785 (ESTABLISHED)
linkerd-t 14178 stevej 353u IPv4 908911358 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33783 (ESTABLISHED)
linkerd-t 14178 stevej 354u IPv4 908911359 0t0 TCP proxy-test-4d:38968->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 355u IPv4 908911360 0t0 TCP proxy-test-4d:55066->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 356u IPv4 908911361 0t0 TCP proxy-test-4d:42192->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 357u IPv4 908911362 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33784 (ESTABLISHED)
linkerd-t 14178 stevej 358u IPv4 908911363 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33790 (ESTABLISHED)
linkerd-t 14178 stevej 359u IPv4 908911364 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33787 (ESTABLISHED)
linkerd-t 14178 stevej 360u IPv4 908911365 0t0 TCP proxy-test-4d:55070->perf-target-4.[62/1893]-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 361u IPv4 908911366 0t0 TCP proxy-test-4d:38976->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 362u IPv4 908911367 0t0 TCP proxy-test-4d:42198->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 363u IPv4 908911368 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33788 (ESTABLISHED)
linkerd-t 14178 stevej 364u IPv4 908911369 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33789 (ESTABLISHED)
linkerd-t 14178 stevej 365u IPv4 908911370 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33774 (ESTABLISHED)
linkerd-t 14178 stevej 366u IPv4 908911371 0t0 TCP proxy-test-4d:55076->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 367u IPv4 908911372 0t0 TCP proxy-test-4d:38982->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 368u IPv4 908911373 0t0 TCP proxy-test-4d:42204->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 369u IPv4 908911374 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33793 (ESTABLISHED)
linkerd-t 14178 stevej 370u IPv4 908911375 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33792 (ESTABLISHED)
linkerd-t 14178 stevej 371u IPv4 908911376 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33786 (ESTABLISHED)
linkerd-t 14178 stevej 372u IPv4 908911377 0t0 TCP proxy-test-4d:55082->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 373u IPv4 908911378 0t0 TCP proxy-test-4d:38988->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 374u IPv4 908911379 0t0 TCP proxy-test-4d:42210->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 375u IPv4 908911380 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33794 (ESTABLISHED)
linkerd-t 14178 stevej 376u IPv4 908911381 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33797 (ESTABLISHED)
linkerd-t 14178 stevej 377u IPv4 908911382 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33798 (ESTABLISHED)
linkerd-t 14178 stevej 378u IPv4 908911383 0t0 TCP proxy-test-4d:42212->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 379u IPv4 908911384 0t0 TCP proxy-test-4d:55090->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 380u IPv4 908911385 0t0 TCP proxy-test-4d:38996->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 381u IPv4 908911386 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33801 (ESTABLISHED)
linkerd-t 14178 stevej 382u IPv4 908911387 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33799 (ESTABLISHED)
linkerd-t 14178 stevej 383u IPv4 908911388 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33795 (ESTABLISHED)
linkerd-t 14178 stevej 384u IPv4 908911389 0t0 TCP proxy-test-4d:38998->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 385u IPv4 908911390 0t0 TCP proxy-test-4d:55096->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 386u IPv4 908911391 0t0 TCP proxy-test-4d:42222->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 387u IPv4 908911392 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33791 (ESTABLISHED)
linkerd-t 14178 stevej 388u IPv4 908911393 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33796 (ESTABLISHED)
linkerd-t 14178 stevej 389u IPv4 908911394 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33800 (ESTABLISHED)
linkerd-t 14178 stevej 390u IPv4 908911395 0t0 TCP proxy-test-4d:55100->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 391u IPv4 908911396 0t0 TCP proxy-test-4d:39006->perf-target-6.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 392u IPv4 908911397 0t0 TCP proxy-test-4d:42228->perf-target-5.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 393u IPv4 908911398 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33802 (ESTABLISHED)
linkerd-t 14178 stevej 394u IPv4 908911399 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33807 (ESTABLISHED)
linkerd-t 14178 stevej 395u IPv4 908911400 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33803 (ESTABLISHED)
linkerd-t 14178 stevej 396u IPv4 908911401 0t0 TCP proxy-test-4d:55106->perf-target-4.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 397u IPv4 908911402 0t0 TCP proxy-test-4d:39012->perf-target-6.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 398u IPv4 908911403 0t0 TCP proxy-test-4d:42234->perf-target-5.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 399u IPv4 908911404 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33805 (ESTABLISHED)
linkerd-t 14178 stevej 400u IPv4 908911405 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33806 (ESTABLISHED)
linkerd-t 14178 stevej 401u IPv4 908911406 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33804 (ESTABLISHED)
linkerd-t 14178 stevej 402u IPv4 908911407 0t0 TCP proxy-test-4d:55112->perf-target-4.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 403u IPv4 908911408 0t0 TCP proxy-test-4d:39018->perf-target-6.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 404u IPv4 908911409 0t0 TCP proxy-test-4d:42240->perf-target-5.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 405u IPv4 908911410 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33808 (ESTABLISHED)
linkerd-t 14178 stevej 406u IPv4 909244034 0t0 TCP proxy-test-4d:44928->perf-target-5.c.buoyant-hosted.internal:http-alt (SYN_SENT)
linkerd-t 14178 stevej 407u sock 0,8 0t0 909253650 protocol: TCP
linkerd-t 14178 stevej 408u IPv4 908911413 0t0 TCP proxy-test-4d:39022->perf-target-6.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 409u IPv4 908911414 0t0 TCP proxy-test-4d:55120->perf-target-4.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 410u IPv4 908911415 0t0 TCP proxy-test-4d:42246->perf-target-5.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 411u IPv4 908911417 0t0 TCP proxy-test-4d:42248->perf-target-5.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 412u IPv4 908911418 0t0 TCP proxy-test-4d:55126->perf-target-4.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 413u IPv4 908911419 0t0 TCP proxy-test-4d:39032->perf-target-6.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej 414u sock 0,8 0t0 909253652 protocol: TCP
linkerd-t 14178 stevej 415u sock 0,8 0t0 909253654 protocol: TCP
linkerd-t 14178 stevej 416u sock 0,8 0t0 909253637 protocol: TCP
linkerd-t 14178 stevej 417u unknown /proc/14178/fd/417 (readlink: No such file or directory)
linkerd-t 14178 stevej 418u sock 0,8 0t0 909253639 protocol: TCP
linkerd-t 14178 stevej 419u sock 0,8 0t0 909252607 protocol: TCP
linkerd-t 14178 stevej 420u sock 0,8 0t0 909253647 protocol: TCP
linkerd-t 14178 stevej 421u sock 0,8 0t0 909253645 protocol: TCP
linkerd-t 14178 stevej 422u sock 0,8 0t0 909253649 protocol: TCP
linkerd-t 14178 stevej 423u IPv4 909235806 0t0 TCP proxy-test-4d:33498->perf-target-6.c.buoyant-hosted.internal:http-alt (SYN_SENT)
linkerd-t 14178 stevej 424u sock 0,8 0t0 909253641 protocol: TCP
linkerd-t 14178 stevej 425u sock 0,8 0t0 909253643 protocol: TCP
linkerd-t 14178 stevej 427u sock 0,8 0t0 909253653 protocol: TCP
linkerd-t 14178 stevej 428u IPv4 909253633 0t0 TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:39840 (ESTABLISHED)
linkerd-t 14178 stevej 429u sock 0,8 0t0 909253635 protocol: TCP
Good news: the same process will happily route requests after being in this state
stevej@netty-test-8:~$ ./slow_cooker_linux_amd64 -host 'default' -totalRequests 1000000 -qps 100 -concurrency 100 http://proxy-test-4d:7474/
# sending 10000 req/s with concurrency=100 to http://proxy-test-4d:7474/ ...
# good/b/f t good% min [p50 p95 p99 p999] max change
2017-04-20T22:02:32Z 99667/0/0 100000 99% 10s 0 [ 2 4 4 74 ] 74 +
2017-04-20T22:02:42Z 100000/0/0 100000 100% 10s 0 [ 2 4 5 13 ] 13 +
2017-04-20T22:02:52Z 100002/0/0 100000 100% 10s 0 [ 2 4 4 14 ] 14
2017-04-20T22:03:02Z 100000/0/0 100000 100% 10s 0 [ 2 4 4 9 ] 9
2017-04-20T22:03:12Z 100000/0/0 100000 100% 10s 0 [ 2 4 4 12 ] 12
2017-04-20T22:03:22Z 100000/0/0 100000 100% 10s 0 [ 2 4 5 11 ] 11
2017-04-20T22:03:32Z 100000/0/0 100000 100% 10s 0 [ 2 4 5 11 ] 11
2017-04-20T22:03:42Z 100002/0/0 100000 100% 10s 0 [ 2 4 5 9 ] 9
2017-04-20T22:03:52Z 100044/0/0 100000 100% 10s 0 [ 2 4 5 9 ] 9
2017-04-20T22:04:02Z 99975/0/0 100000 99% 10s 0 [ 2 4 5 9 ] 9
2017-04-20T22:04:12Z 100010/0/0 100000 100% 10s 0 [ 2 4 5 8 ] 8
FROM TO #REQUESTS
0 2 167367
2 8 932083
8 32 196
32 64 41
64 128 13
128 256 0
256 512 0
512 1024 0
1024 4096 0
4096 16384 0