NuGet
[NuGet.org Bug]: api.nuget.org is slow from some geographic locations
Impact
It's more difficult to complete my work
Describe the bug
api.nuget.org is slow from some geographic locations, including where I'm based (Australia).
To test this, I created DigitalOcean Droplets in different regions and ran tests using ApacheBench.
This table shows the time taken to complete 100 requests for a resource on api.nuget.org, using HTTP Keep Alive.
| Location | Time taken for tests (s) | Time taken for tests, relative to New York |
|---|---|---|
| New York | 3.04 | 1.00 |
| San Francisco | 5.627 | 1.85 |
| London | 10.518 | 3.46 |
| Singapore | 22.089 | 7.27 |
When using NuGet Package Manager UI, the Updates and Installed tabs can make several hundred API requests. This combined with the slow response time degrades the user experience for people in certain locations.
From my home machine, I'm seeing times comparable to Singapore in the table above.
Detailed results:
New York
This is ApacheBench, Version 2.3 <$Revision: 1901567 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking api.nuget.org (be patient).....done
Server Software: Windows-Azure-Blob/1.0
Server Hostname: api.nuget.org
Server Port: 443
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key: ECDH P-256 256 bits
TLS Server Name: api.nuget.org
Document Path: /v3/registration5-gz-semver2/microsoft.aspnetcore.mvc.testing/index.json
Document Length: 18694 bytes
Concurrency Level: 1
Time taken for tests: 3.040 seconds
Complete requests: 100
Failed requests: 0
Keep-Alive requests: 100
Total transferred: 1955600 bytes
HTML transferred: 1869400 bytes
Requests per second: 32.89 [#/sec] (mean)
Time per request: 30.402 [ms] (mean)
Time per request: 30.402 [ms] (mean, across all concurrent requests)
Transfer rate: 628.18 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.2 0 12
Processing: 28 30 7.9 29 105
Waiting: 28 30 7.9 29 105
Total: 28 30 9.1 29 117
Percentage of the requests served within a certain time (ms)
50% 29
66% 29
75% 30
80% 30
90% 31
95% 32
98% 48
99% 117
100% 117 (longest request)
San Francisco
This is ApacheBench, Version 2.3 <$Revision: 1901567 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking api.nuget.org (be patient).....done
Server Software: Windows-Azure-Blob/1.0
Server Hostname: api.nuget.org
Server Port: 443
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key: ECDH P-256 256 bits
TLS Server Name: api.nuget.org
Document Path: /v3/registration5-gz-semver2/microsoft.aspnetcore.mvc.testing/index.json
Document Length: 18694 bytes
Concurrency Level: 1
Time taken for tests: 5.627 seconds
Complete requests: 100
Failed requests: 0
Keep-Alive requests: 100
Total transferred: 1955600 bytes
HTML transferred: 1869400 bytes
Requests per second: 17.77 [#/sec] (mean)
Time per request: 56.272 [ms] (mean)
Time per request: 56.272 [ms] (mean, across all concurrent requests)
Transfer rate: 339.38 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.0 0 10
Processing: 55 56 2.0 56 75
Waiting: 55 56 2.0 56 75
Total: 55 56 2.4 56 75
Percentage of the requests served within a certain time (ms)
50% 56
66% 56
75% 56
80% 56
90% 57
95% 58
98% 70
99% 75
100% 75 (longest request)
London
This is ApacheBench, Version 2.3 <$Revision: 1901567 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking api.nuget.org (be patient).....done
Server Software: Windows-Azure-Blob/1.0
Server Hostname: api.nuget.org
Server Port: 443
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key: ECDH P-256 256 bits
TLS Server Name: api.nuget.org
Document Path: /v3/registration5-gz-semver2/microsoft.aspnetcore.mvc.testing/index.json
Document Length: 18694 bytes
Concurrency Level: 1
Time taken for tests: 10.518 seconds
Complete requests: 100
Failed requests: 0
Keep-Alive requests: 100
Total transferred: 1955600 bytes
HTML transferred: 1869400 bytes
Requests per second: 9.51 [#/sec] (mean)
Time per request: 105.182 [ms] (mean)
Time per request: 105.182 [ms] (mean, across all concurrent requests)
Transfer rate: 181.57 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.9 0 9
Processing: 100 105 29.7 101 394
Waiting: 100 105 29.7 101 394
Total: 100 105 30.6 101 403
Percentage of the requests served within a certain time (ms)
50% 101
66% 101
75% 101
80% 101
90% 102
95% 113
98% 140
99% 403
100% 403 (longest request)
Singapore
This is ApacheBench, Version 2.3 <$Revision: 1901567 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking api.nuget.org (be patient).....done
Server Software: Windows-Azure-Blob/1.0
Server Hostname: api.nuget.org
Server Port: 443
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key: ECDH P-256 256 bits
TLS Server Name: api.nuget.org
Document Path: /v3/registration5-gz-semver2/microsoft.aspnetcore.mvc.testing/index.json
Document Length: 18694 bytes
Concurrency Level: 1
Time taken for tests: 22.089 seconds
Complete requests: 100
Failed requests: 0
Keep-Alive requests: 100
Total transferred: 1955600 bytes
HTML transferred: 1869400 bytes
Requests per second: 4.53 [#/sec] (mean)
Time per request: 220.891 [ms] (mean)
Time per request: 220.891 [ms] (mean, across all concurrent requests)
Transfer rate: 86.46 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.8 0 8
Processing: 214 221 63.7 214 851
Waiting: 214 221 63.7 214 851
Total: 214 221 64.5 214 860
Percentage of the requests served within a certain time (ms)
50% 214
66% 214
75% 215
80% 215
90% 215
95% 216
98% 219
99% 860
100% 860 (longest request)
Repro Steps
- Install ApacheBench
apt-get install -y apache2-utils
- Run tests
ab -n 100 -k https://api.nuget.org/v3/registration5-gz-semver2/microsoft.aspnetcore.mvc.testing/index.json
Expected Behavior
Response times should not vary so dramatically with geographic location.
Screenshots
No response
Additional Context and logs
After further testing, I found that this does not apply to https://api.nuget.org/v3/index.json.
For /v3/index.json, the Server response header value (ECAcc (mbw/47ED)) shows that it is being served from an Edgio PoP, while for other requests the value is Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0.
Hey @mjolka, thanks for the detailed report. tl;dr: this is a known issue but we don't have a date we can share of when it will be improved. That being said, on the client side, we've increase parallelism in VS per https://github.com/NuGet/Home/issues/11923 (which I see you've also originally posted, thank you for the focus on performance!).
This is a known issue primarily affecting the performance of the Package Manager UI, as you've stated. There are other JSON endpoints with the same symptom you're seeing (slower latency across geos) but, based on end-to-end performance tests (e.g. total dotnet restore duration) the slower response time does not significantly impact the overall experience. Unfortunately, this disconnect between endpoint latency and overall operation latency is not the case with the package metadata endpoint (/v3/registration5-gz-semver2-like URLs) and the lower latency across geos indeed impacts the responsiveness of the UI (again, confirming what you're saying).
Generally speaking, for the "api.nuget.org" domain, we have a variety of caching mechanisms and several geographic locations backing the CDN nodes which serve requests. For example, we have a dedicated storage CDN and storage backend in China yielding superior performance in this case. The "https://api.nuget.org/v3/index.json" URL is highly cached at the CDN level which is why the performance is good across geos. Similarly, .nupkg and .nuspec URLs are highly cached too.
We've been unable to cache JSON endpoints in most cases due to concerns about data consistency across multiple URLs that the client interacts with in a session but it's an area we're experimenting with for both performance and availability reasons. I don't have an ETA for you, however.
For the particular user scenario of the Browse tab in the PM UI, we have https://github.com/NuGet/NuGetGallery/issues/7297 planned which will alleviate the need for so many HTTP requests, thus reducing the impact of lower latency across geos. Unfortunately, this work will not help the Installed or Updates tab which need to pull information about a set of IDs known only to the client (compared to the Browse tab where the set of IDs is provided by the server).
Thanks for the high-quality issue report. I'll leave this issue open since it is indeed a problem but I can't provide you a timeline of when it will get better.
@joelverhagen has there been any movement on the CDN being slow in certain regions?
I'm seeing unreasonable slowness in the UK getting this package: https://api.nuget.org/v3-flatcontainer/syncfusion.blazor.themes/27.2.2/syncfusion.blazor.themes.27.2.2.nupkg It's not just slow, but unbelievably slow (less than 5kb/s on an 8gig connection). I've also switched UK locations via a vpn, but no real difference, the downloads speed along for the first bit of cached (at the CDN) data, then come at treacle speed.
It's also like the first 8mb has been cached (by my previous attempts) but the remainder is coming at 15kb/s with pauses. I'm up to 9mb now, but you can imagine how long this has taken. Other files didn't appear to have the same issue when I installed syncfusion.blazor.inputs, presumably due to smaller sizes. Jump on a VPN and get into the UK, try to download that file, I've cancelled mine so the cache doesn't finish the file. It might be worth noting I'm using 1.1.1.1(cloudflare) for DNS on my home router.
Hi! @tyeth My apologies for the inconvenience! Multiple customers reported that CDN is slow in UK. We have switched the traffic to another CDN service. Feel free to reach out to support (https://www.nuget.org/policies/Contact) if the issue is still ongoing.
