packagecloud.io : HTTP Status 429 : "Bandwidth or Storage Limit Exceeded"

[grempe]

I was running an installations script that I have been running without error recently and it suddenly failed today for any Varnish use of apt-get update.

The errors are:

$ sudo apt-get update
Hit:1 http://us-east4-a.gce.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://us-east4-a.gce.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:3 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Get:4 http://us-east4-a.gce.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Hit:5 http://archive.canonical.com/ubuntu xenial InRelease
Ign:6 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial InRelease
Ign:7 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial Release
Ign:8 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Sources
Ign:9 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main amd64 Packages
Ign:10 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main all Packages
Ign:11 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en_US
Ign:12 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en
Ign:8 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Sources
Ign:9 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main amd64 Packages
Ign:10 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main all Packages
Ign:11 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en_US
Ign:12 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en
Ign:8 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Sources
Ign:9 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main amd64 Packages
Ign:10 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main all Packages
Ign:11 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en_US
Ign:12 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en
Ign:8 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Sources
Ign:9 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main amd64 Packages
Ign:10 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main all Packages
Ign:11 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en_US
Ign:12 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en
Ign:8 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Sources
Ign:9 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main amd64 Packages
Ign:10 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main all Packages
Ign:11 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en_US
Ign:12 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en
Err:8 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Sources
  404  Not Found
Ign:9 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main amd64 Packages
Ign:10 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main all Packages
Ign:11 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en_US
Ign:12 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en
Fetched 306 kB in 2s (117 kB/s)
Reading package lists... Done
W: The repository 'https://packagecloud.io/varnishcache/varnish52/ubuntu xenial Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Failed to fetch https://packagecloud.io/varnishcache/varnish52/ubuntu/dists/xenial/main/source/Sources  404  Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.

I tried to manually retrieve some of the specified files and the following error is returned:

$ curl -i https://packagecloud.io/varnishcache/varnish52/ubuntu/dists/xenial/main/source/Sources
HTTP/1.1 429 Too Many Requests
Server: nginx
Date: Sun, 01 Oct 2017 00:21:32 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 35
Connection: keep-alive
Status: 429 Too Many Requests
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Cache-Control: no-cache
X-Request-Id: 95bf477d-c199-45c0-88ae-521802bc50a7
X-Runtime: 0.006280
Strict-Transport-Security: max-age=31536000

Bandwidth or Storage Limit Exceeded

HTTP Status Code 429 and Bandwidth or Storage Limit Exceeded. All installs via packagecloud.io are apparently broken for Varnish at the moment. This is a rather urgent issue.

[gquintard]

Can't reproduce here, did we go a bit hard on the download rate?

-- Guillaume Quintard

On Sun, Oct 1, 2017 at 2:23 AM, Glenn Rempe [email protected] wrote:

I was running an installations script that I have been running without error recently and it suddenly failed today for any Varnish use of apt-get update.

The errors are:

$ sudo apt-get update Hit:1 http://us-east4-a.gce.clouds.archive.ubuntu.com/ubuntu xenial InRelease Get:2 http://us-east4-a.gce.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB] Get:3 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB] Get:4 http://us-east4-a.gce.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB] Hit:5 http://archive.canonical.com/ubuntu xenial InRelease Ign:6 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial InRelease Ign:7 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial Release Ign:8 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Sources Ign:9 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main amd64 Packages Ign:10 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main all Packages Ign:11 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en_US Ign:12 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en Ign:8 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Sources Ign:9 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main amd64 Packages Ign:10 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main all Packages Ign:11 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en_US Ign:12 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en Ign:8 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Sources Ign:9 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main amd64 Packages Ign:10 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main all Packages Ign:11 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en_US Ign:12 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en Ign:8 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Sources Ign:9 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main amd64 Packages Ign:10 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main all Packages Ign:11 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en_US Ign:12 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en Ign:8 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Sources Ign:9 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main amd64 Packages Ign:10 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main all Packages Ign:11 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en_US Ign:12 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en Err:8 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Sources 404 Not Found Ign:9 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main amd64 Packages Ign:10 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main all Packages Ign:11 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en_US Ign:12 https://packagecloud.io/varnishcache/varnish52/ubuntu xenial/main Translation-en Fetched 306 kB in 2s (117 kB/s) Reading package lists... Done W: The repository 'https://packagecloud.io/varnishcache/varnish52/ubuntu xenial Release' does not have a Release file. N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use. N: See apt-secure(8) manpage for repository creation and user configuration details. E: Failed to fetch https://packagecloud.io/varnishcache/varnish52/ubuntu/dists/xenial/main/source/Sources 404 Not Found E: Some index files failed to download. They have been ignored, or old ones used instead.

I tried to manually retrieve some of the specified files and the following error is returned:

$ curl -i https://packagecloud.io/varnishcache/varnish52/ubuntu/dists/xenial/main/source/Sources HTTP/1.1 https://packagecloud.io/varnishcache/varnish52/ubuntu/dists/xenial/main/source/SourcesHTTP/1.1 429 Too Many Requests Server: nginx Date: Sun, 01 Oct 2017 00:21:32 GMT Content-Type: text/html;charset=utf-8 Content-Length: 35 Connection: keep-alive Status: 429 Too Many Requests X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN Cache-Control: no-cache X-Request-Id: 95bf477d-c199-45c0-88ae-521802bc50a7 X-Runtime: 0.006280 Strict-Transport-Security: max-age=31536000

Bandwidth or Storage Limit Exceeded

HTTP Status Code 429 and Bandwidth or Storage Limit Exceeded. All installs via packagecloud.io are apparently broken for Varnish at the moment. This is a rather urgent issue.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/varnishcache/pkg-varnish-cache/issues/89, or mute the thread https://github.com/notifications/unsubscribe-auth/ADmgKbQjR4MoTfDKYPR4suejFZJIlsheks5sntuYgaJpZM4PpvNp .

[grempe]

Seems resolved now.

[espebra]

I'll just shed some light here. Packagecloud had issues with traffic accounting that lead to certain repositories being wrongfully blocked for a couple of hours. The error message you reported here was the result during this period. It should not happen again.

[rbran100]

@espebra @grempe This is happening again.

[espebra]

@rbran100 Thank you for the heads up, I'll look into it. Btw it works now.

[damienpeerbolte]

The same problem is happening again

[gquintard]

@damienpeerbolte thanks for the heads up, looks like this repository is pretty popular I'll reopen and so we can monitor the situation this week, please add any observations you may have