Ignored HTTP status codes that raise exceptions are reported as errors for parent Active Support Cache span

[caramcc]

Is your feature request related to a problem? Please describe. Active Support cache spans report errors for HTTP requests that are not considered errors by the ddtrace instrumentation for the HTTP client when the HTTP response is raised as an exception.

For example, the following block of code results in an Active Support cache span with an error, even when error_status_codes is configured so that 4xx-level responses are not treated as errors and the span for the http client does not report an error:

Rails.cache.fetch("non_existent_key") do
  api_request_that_returns_http_403_and_raises_it_as_an_exception_for_some_reason
end

My understanding is that this is due to the ActiveSupport cache instrumentation's Span object raising the generated execption while the HTTP library code only annotates the span with the error if the request code matches, regardless of any exceptions that are raised.

Questions:

• Is my understanding of the cause of this issue correct?
• Would it be feasible to add a configuration setting to the HTTP tracers that would allow these exceptions to be discarded so they aren't confusingly raised by the Active Support Cache tracer?

Describe the goal of the feature In my case, I'd like the error to also be ignored by the Active Support Cache span — having these ignorable responses surfaced as active support cache errors is confusing and leads to higher span ingestion than we actually want or need.

Describe alternatives you've considered Trying to rescue the exceptions as they're raised in our application code — in our case this would be difficult to the point of being infeasible, since this code often comes from 3rd-party gems for 3rd party APIs, and can't easily enumerate all of these random exceptions everywhere these HTTP requests are made.

Additional context Our applications make a lot of external HTTP requests that are expected to have a relatively high failure rate, so we configure error_status_codes for the various HTTP libraries that we use. (For additional context, see my previous feature request that ultimately resulted in the implementation of the error_status_codes option: https://github.com/DataDog/dd-trace-rb/issues/2492).

Thanks in advance!