buildpacks-ci
buildpacks-ci copied to clipboard
cloudfoundry
getting error "read: connection reset by peer" while downloading bpack deps
we are having recurring issues while downloading buildpacks dependencies from 2 specific regions : us-south, us-east our logs indicates a connection reset issue as following examples :
2023-08-22 06:02:04 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-0-1/STG] Download [https://buildpacks.cloudfoundry.org/dependencies/dotnet-sdk/dotnet-sdk_6.0.408_linux_x64_any-stack_fad207e0.tar.xz] 2023-08-22 06:02:06 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-0-1/STG] **ERROR** Unable to install Dotnet SDK: read tcp 10.244.66.125:41976->104.118.246.84:443: read: connection reset by peer
2023-08-22 06:02:04 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-0-1/STG] Download [https://buildpacks.cloudfoundry.org/dependencies/dotnet-sdk/dotnet-sdk_6.0.408_linux_x64_any-stack_fad207e0.tar.xz]
2023-08-22 06:02:06 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-0-1/STG] **ERROR** Unable to install Dotnet SDK: read tcp 10.244.66.125:41976->104.118.246.84:443: read: connection reset by peer
2023-08-22 05:37:52 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-3-2/STG] Download [https://buildpacks.cloudfoundry.org/dependencies/dotnet-sdk/dotnet-sdk_6.0.407_linux_x64_any-stack_f1497b98.tar.xz]
2023-08-22 05:37:55 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-3-2/STG] **ERROR** Unable to install Dotnet SDK: read tcp 10.245.203.234:36272->104.118.246.84:443: read: connection reset by peer
2023-08-22 06:22:02 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-1-1-0/STG] Download [https://buildpacks.cloudfoundry.org/dependencies/python/python_3.10.10_linux_x64_cflinuxfs3_73c718f0.tgz]
2023-08-22 06:22:03 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-1-1-0/STG] **ERROR** Unable to bootstrap python: read tcp 10.244.253.84:46256->104.118.246.84:443: read: connection reset by peer
after some more in deep investigations we noticed that your akamai service is limiting the download rate to 10MB returning error in case the download speed is higher but this is specific for some ip addresses because when we connect from another availability zone the limit is to 100MB
in our case connecting from those US-EAST regional ip , AZ1 and AZ2 have the lower limit while AZ3 has the higher
AZ1 52.116.125.68 10MB/s
AZ2 169.59.191.31 10MB/s
AZ3 150.239.226.24 100MB/s
also connecting from US-SOUTH regional ip , AZ1 and AZ2 have the lower limit while AZ3 has the higher
AZ1 150.240.65.245 10MB/s
AZ2 150.239.168.150 10MB/s
AZ3 169.48.95.93 100MB/s
fyi this is the command we use for test from US-SOUTH
export cdn_ip=104.118.246.84; curl --limit-rate 50M -o x.xz https://buildpacks.cloudfoundry.org/dependencies/dotnet-sdk/dotnet-sdk_6.0.407_linux_x64_any-stack_f1497b98.tar.xz --resolve buildpacks.cloudfoundry.org:443:$cdn_ip
replace cdn_ip=104.96.236.92 for US-EAST
summarising we noticed a rate_limit to 10MB/s when connecting
to
104.96.236.92
from
52.116.125.68
169.59.191.31
and
to
104.118.246.84
from
150.240.65.245
150.239.168.150
I perform the trace route command in 3 different cells in different our us-south env in AZ1,AZ2, and AZ3 and all of them I got the similar output
traceroute 104.118.246.84
traceroute to 104.118.246.84 (104.118.246.84), 30 hops max, 60 byte packets
1 x.x.x.x (x.x.x.x) 0.118 ms 0.087 ms 0.065 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
ping 104.118.246.84
PING 104.118.246.84 (104.118.246.84) 56(84) bytes of data.
64 bytes from 104.118.246.84: icmp_seq=1 ttl=52 time=6.96 ms
64 bytes from 104.118.246.84: icmp_seq=2 ttl=52 time=6.82 ms
64 bytes from 104.118.246.84: icmp_seq=3 ttl=52 time=6.88 ms
64 bytes from 104.118.246.84: icmp_seq=4 ttl=52 time=6.80 ms
^C
--- 104.118.246.84 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 6.796/6.862/6.956/0.061 ms
I also installed mtr in one cell and this is the output:
diego-cell-us-south-1-0-0 (x.x.x.x) -> 104.118.246.84 (104.118.246.84) 2023-08-22T19:23:40+0000
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. x.x.x.x 0.0% 24 0.1 0.1 0.1 0.1 0.0
2. (waiting for reply)
3. (waiting for reply)
4. (waiting for reply)
5. (waiting for reply)
6. (waiting for reply)
7. (waiting for reply)
8. (waiting for reply)
9. (waiting for reply)
10. (waiting for reply)
11. (waiting for reply)
12. (waiting for reply)
13. (waiting for reply)
14. (waiting for reply)
15. 104.118.246.84 0.0% 23 6.9 6.9 6.8 7.0 0.0
Same thing in our us-east env:
# traceroute 104.96.236.92
traceroute to 104.96.236.92 (104.96.236.92), 30 hops max, 60 byte packets
1 x.x.x.x (x.x.x.x) 0.192 ms 0.041 ms 0.019 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
ping 104.96.236.92
PING 104.96.236.92 (104.96.236.92) 56(84) bytes of data.
64 bytes from 104.96.236.92: icmp_seq=1 ttl=57 time=1.50 ms
64 bytes from 104.96.236.92: icmp_seq=2 ttl=57 time=1.62 ms
64 bytes from 104.96.236.92: icmp_seq=3 ttl=57 time=1.55 ms
64 bytes from 104.96.236.92: icmp_seq=4 ttl=57 time=1.59 ms
^C
--- 104.96.236.92 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 1.498/1.562/1.617/0.045 ms
We worked to create a new machine on a Cloud IaaS out of any VPC and tested with same results. We also used some other VMs in other Infrastructure and still same issue. so it is not limited to those source IPs i pasted before. Can you pls ask CDN team if there is somehow a rate limit and to test the curl command being sure the download speed goes over 50MB/s (as an example) ?
another wave is currently happening on us-south region.
2023-08-23 12:49:27 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-4-1/STG] Download [https://buildpacks.cloudfoundry.org/dependencies/dotnet-sdk/dotnet-sdk_6.0.408_linux_x64_any-stack_fad207e0.tar.xz]
2023-08-23 12:49:27 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-4-1/STG] -----> Installing dotnet-sdk 6.0.408
2023-08-23 12:49:29 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-4-1/STG] **ERROR** Unable to install Dotnet SDK: read tcp 10.247.83.10:35410->104.118.246.84:443: read: connection reset by peer
2023-08-23 12:49:29 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-4-1/STG] Failed to compile droplet: Failed to run all supply scripts: exit status 15
2023-08-23 12:49:29 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-4-1/STG] Exit status 223
