Warnings that occur when HTTP2 downloads are throttled are a little bit aggressive

[thoughtpolice]

https://github.com/thoughtpolice/buck2/actions/runs/5418361387/jobs/9850380998

If you check the Build Buck2 with Buck2 step, you'll a duplicate number of rather aggressive annoying — I believe these originate from being too aggressive when downloading packages from https://crates.io, and getting rate limited:

WARN buck2_common::http::retries: Retrying a HTTP error after 2 seconds: HTTP Client error: HTTP: Error sending request: http2 error: stream error received: refused stream before processing any application logic

However, it's actually a bit worse when superconsole is in use in your terminal — here it just uses a more typical log stream that streams once — because it gets spammed a bunch during refresh of the active targets being built. Looks like it happens dozens of times in Windows Terminal for me, at least. I can attach a small asciinema of this but it should be easy to reproduce, I think.

Honestly, I don't think this warning as it stands now is really that important, a little rate limiting is expected if you're doing a fresh build with no cache, I think. You're downloading like 100+ dependent crates pretty quickly in this scenario. The download phase of the crates being extended by a second or two is probably OK if you end up getting a big set of them in batches, it may only add a few seconds to the initial download actions.

I think better logic would be if Buck2:

• Downloads with typical exponential backoff during throttling (I assume this is the case already), and
• Only throws an error after a number of retries results in, say, a backoff window of 60 seconds or more occurs.

60 seconds is arbitrary but seems like it's egregious enough to make that a big orange warning. A hair you'd have to split is whether or not you would only do one warning or just warn on every single action. Currently it looks like the warning occurs for every action that gets throttled, but maybe you only want one warning per HTTP domain; e.g. if two download_file actions to download two separate files from http://foo.example.com get rate limited, only report once for the foo.example.com domain — and if another download_file call to http://bar.example.com also gets rate limited, report that, so you'd only have two warnings total.

[nickgerace]

This is really pummeling us, our users, and it is affecting our ability to share logs. I'm coming into this a bit late (though we've been dealing with it since we started using buck2 in spring 2023), but I'm looking at https://github.com/facebook/buck2/issues/332 and https://github.com/facebook/buck2/pull/342 and I wonder if it may be possible just to adjust/toggle this warning log and/or its timing in isolation.

Here is a snippet (this happens for most-to-all third-party crates in our monorepo):

[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.459388Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/ouroboros_macro/0.18.5/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.459391Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/unicase/2.8.1/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.458555Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/static_assertions/1.1.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.459400Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/sqlx/0.8.3/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.459404Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/fixedbitset/0.4.2/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.458556Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/nanorand/0.7.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.459413Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/secrecy/0.8.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.459413Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/num_cpus/1.16.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.459419Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/rand_chacha/0.9.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.457977Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/color-spantrace/0.2.1/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.459430Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/superboring/0.1.4/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.459433Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/peeking_take_while/0.1.2/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.459440Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/ordered-float/2.10.1/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.458627Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/aws-sigv4/1.3.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.458626Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/ct-codecs/1.1.3/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.459463Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/crypto-bigint/0.5.5/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.459467Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/rustls-native-certs/0.8.1/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.458662Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/tracing-error/0.2.1/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.459481Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/refinery-core/0.8.16/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.459478Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/hashbrown/0.13.2/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.459491Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/ahash/0.8.11/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.458033Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/icu_properties/1.5.1/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.459507Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/array-util/1.0.2/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.458684Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/humantime/2.2.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.459527Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/ed25519-compact/2.1.1/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.458688Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/aws-smithy-http-client/1.0.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.458041Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/option-ext/0.2.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.458723Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/quinn/0.11.6/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.601-04:00] 2025-03-21T15:21:53.458736Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/rand_core/0.9.3/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458736Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/k256/0.13.4/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458797Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/zstd/0.13.3/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458122Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/rfc6979/0.4.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458043Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/hashbrown/0.15.2/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458830Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/futures-timer/3.0.3/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458858Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/ouroboros/0.18.5/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458892Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/ff/0.13.1/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458259Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/small_ctor/0.1.2/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458892Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/aws-smithy-json/0.61.3/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458895Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/base64/0.21.7/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458902Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/derive_utils/0.15.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458902Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/rand_chacha/0.2.2/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458300Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/postgres-types/0.2.9/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458912Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/darrentsung_debug_parser/0.3.1/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458935Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/p384/0.13.1/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458322Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/elliptic-curve/0.13.8/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458955Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/itertools/0.13.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.457703Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/aws-types/1.3.6/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458998Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/hash32/0.2.1/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458998Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/foyer/0.14.1/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459020Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/foyer-common/0.14.1/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458379Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/reqwest-eventsource/0.6.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459131Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/filetime/0.2.25/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459151Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/embedded-io/0.4.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459155Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/serde-value/0.7.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459194Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/utf8-width/0.1.7/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459195Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/tokio-util/0.7.14/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.456710Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/x509-cert/0.2.5/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459246Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/tracing-tunnel/0.1.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459244Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/monostate/0.1.14/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458472Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/alloc-no-stdlib/2.0.4/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.457892Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/futures-sink/0.3.31/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459314Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/hyper/0.14.32/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458489Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/arrayvec/0.7.6/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458501Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/is-terminal/0.4.16/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459351Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/bincode/1.3.3/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459351Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/smallstr/0.3.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459353Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/zerocopy/0.7.35/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459356Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/atomic_refcell/0.1.13/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458528Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/ordered_hash_map/0.4.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459407Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/crc/3.2.1/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458578Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/futures/0.3.31/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459438Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/ecdsa/0.16.9/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459446Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/xattr/1.5.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.457986Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/anstyle-query/1.1.2/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459455Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/aws-sdk-sts/1.63.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.458668Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/rustc-demangle/0.1.24/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic
[2025-03-21T11:21:53.602-04:00] 2025-03-21T15:21:53.459518Z  WARN buck2_http::retries: Retrying a HTTP error after 2 seconds: Error performing http_head request: HTTP: Error sending request to https://static.crates.io/crates/y-sync/0.4.0/download: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic

[thoughtpolice]

#342 basically stalled out due to some extra changes that needed to be made, and I never got back to it. Honestly, I just ended up deleting this warning from my fork of buck2 (buck2_http: remove an annoying warning). It is not actionable for a user in any way and mostly just annoying right now, but I understand why the original PR was rejected.

(Note that anyone is free to use my fork; it has several nice patches that haven't gone upstream yet, which you might be interested in. main is rebased semi-frequently and I don't really provide support for it at the moment, but you could probably use it without too much problem. I am using it in lieu of some patches stalling out and mostly can only resubmit them on my own time, which is scarce right now.)

[nickgerace]

Thank you for the context @thoughtpolice. We may explore using a similar fork.

That being said, I do worry that this is something that affects all non-Meta buck2 users and is invisible to users within Meta.

[Ralith]

Would throttling outbound requests (e.g. #922) be a more graceful solution here? It's interesting to note that cargo itself is routinely used to download hundreds of crates and doesn't seem to have trouble, so presumably there's a happy path here.

[Ralith]

Digging around some, it looks like the root issue here is that the hyper HTTP/2 client optimistically (legally) assumes that crates.io will accept unlimited concurrent requests, while the actual limit (for that particular server) is 128. https://github.com/hyperium/h2/issues/731 discusses this and introduces configuration and a more conservative default in hyper 1.3, but buck2 is still on 0.14, which has not received a backport. I think the best solution would be to either update buck2 to hyper 1.3+, or backport the change to hyper 0.14. I'm assuming the former hasn't happened yet because it's hard, so I may have a go at the latter.

[cormacrelf]

I also frequently get hard errors in similar circumstances when buck drives remote execution, presumably because the nativelink RBE backend is enforcing h2 stream limits too. Do we think updating hyper will fix these as well?

[2025-06-04T00:15:03.993+00:00] Internal error (stage: remote_call_error): Remote Execution Error on Execute with digest ff48d0ab3815764b808f11dd0ba70487dc8259c5e17134c3c1aefed5d6164ef7:141 for ReSession GRPC-SESSION-ID
[2025-06-04T00:15:03.993+00:00] Error: (Failed to start remote execution: status: Unavailable, message: "h2 protocol error: http2 error: stream error received: refused stream before processing any application logic", details: [], metadata: MetadataMap { headers: {} }: transport error: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic)
[2025-06-04T00:15:03.993+00:00] Remote action digest: 'ff48d0ab3815764b808f11dd0ba70487dc8259c5e17134c3c1aefed5d6164ef7:141'
[2025-06-04T00:15:03.993+00:00] Stdout: <empty>
[2025-06-04T00:15:03.993+00:00] Stderr: <empty>

[Ralith]

The error message has a different prefix. I wonder if that's going through tonic? I think tonic still goes through hyper, so it'll probably get fixed regardless. You might also try setting Endpoint::concurrency_limit in the mean time.

[Ralith]

Now that #922 is merged, these errors can now be prevented by adding something like the following to your .buckconfig:

[http]
max_concurrent_requests = 32

#990 should resolve the issue for good regardless of configuration.