How to get HTTP gem's current User-Agent easily? (HTTP gem updates change the User-Agent header, annoyingly breaking webmocks.)

[KelseyDH]

I recently upgraded my http gem version recently, and the upgrade broke lots of webmock tests recording our outgoing http calls, simply because the User-Agent header of my http gem had changed from http.rb/4.1.1 to http.rb/4.2.0

Needing to manually update web mocks between HTTP gem updates is cumbersome, and what would be nice is if the http gem had a class level method like HTTP.user_agent to get the default User-Agent headers the http gem uses, so that web mocks could be written to future-proof them from http gem version updates.

E.g. in my tests I have webmock testing code like this:

  def stub_confirmations_api(response_code)
    stub_request(:post, "#{my_url}/confirmations").
      with(
        headers: {
          "Auth-Id": ENV.fetch("AUTH_ID"),
          "Auth-Token": ENV.fetch("AUTH_TOKEN"),
          "Connection": 'close',
          "Host": 'localhost',
          "User-Agent": 'http.rb/4.1.1'
        }
      ).to_return(status: response_code, body: nil, headers: {})
  end

When I updated my http gem's version, "User-Agent": 'http.rb/4.1.1' needed to be manually updated to "User-Agent": 'http.rb/4.2.0':

  def stub_confirmations_api(response_code)
    stub_request(:post, "#{my_url}/confirmations").
      with(
        headers: {
          "Auth-Id": ENV.fetch("AUTH_ID"),
          "Auth-Token": ENV.fetch("AUTH_TOKEN"),
          "Connection": 'close',
          "Host": 'localhost',
          "User-Agent": 'http.rb/4.2.0'
        }
      ).to_return(status: response_code, body: nil, headers: {})
  end

The way these webmock tests fail is not easy to debug, and this maintenance burden could be avoided if for User-Agent I could just invoke a User-Agent method from the http gem to get its default user-agent (e.g. HTTP.user_agent) every time. E.g.:

  def stub_confirmations_api(response_code)
    stub_request(:post, "#{my_url}/confirmations").
      with(
        headers: {
          "Auth-Id": ENV.fetch("AUTH_ID"),
          "Auth-Token": ENV.fetch("AUTH_TOKEN"),
          "Connection": 'close',
          "Host": 'localhost',
          "User-Agent": HTTP.user_agent
        }
      ).to_return(status: response_code, body: nil, headers: {})
  end

so that upgrading this gem never breaks webmocks for no other reason than the fact that the gem's version was updated.

[tarcieri]

Instead of:

"User-Agent": 'http.rb/4.2.0'

you can do:

"User-Agent": "http.rb/#{HTTP::VERSION}"

There is no expectation of ever changing this format. I agree a method would be nice, but the above works today and is unlikely to ever break in the future.

[KelseyDH]

Nice! This is a good fix for now. Having a method to be able to set or get HTTP.default_headers as a hash would also be nice. Wouldn't be too useful for people configuring their HTTP gem calls heavily, but could be nice for more basic use cases.

[mikegee]

👍

Having a method to be able to set or get HTTP.default_headers as a hash would also be nice.

I agree with the sentiment in this request. In my apps, I have a method to get a new client that always sets the user-agent. Here's an example:

    def http_client
      HTTP
        .headers('User-Agent' => "MyAppName #{VERSION}",
                 'Accept'     => 'application/json')
        .timeout(:global, connect: TIMEOUT, write: TIMEOUT, read: TIMEOUT)
    end

I don't think it's appropriate for requests from my application out to the wider internet to have a user-agent from the underlying library.

[ixti]

For simplicity of those who need to fetch default user agent, we should expose it as a constant IMO:

class HTTP::Client
  DEFAULT_USER_AGENT = "http.rb/#{HTTP::VERSION}"
end

[KelseyDH]

For simplicity of those who need to fetch default user agent, we should expose it as a constant IMO:

class HTTP::Client
  DEFAULT_USER_AGENT = "http.rb/#{HTTP::VERSION}"
end

I like this idea. Something like HTTP::DEFAULT_USER_AGENT would be concise, yet clear enough that those modifying their User-Agent headers would know it doesn't apply to them.

[tarcieri]

Placing it in HTTP::Client probably makes more sense as the HTTP namespace tends to get a bit polluted

[KelseyDH]

Placing it in HTTP::Client probably makes more sense as the HTTP namespace tends to get a bit polluted

True, HTTP is a common namespace. HTTP::Client::DEFAULT_USER_AGENT works. Though it seems that if HTTP::VERSION is safe HTTP::DEFAULT_USER_AGENT would be too? I'm easy either way but it feels gem default values should be accessible at a gem's top-level.

If we stick with the HTTP::Client namespace, HTTP::Client.default_user_agent might be a cleaner way of doing it via a mattr_accessor:

class HTTP::Client

  DEFAULT_USER_AGENT = "http.rb/#{HTTP::VERSION}"
  mattr_accessor :default_user_agent, default: DEFAULT_USER_AGENT

end

[ixti]

mattr_accessor is a Rails method. And I don't think it's correct to override user agent of default client.

[ixti]

Also, I feel myself a bit stupid, but default user agent is long-time available as HTTP::Request::USER_AGENT

[ixti]

Probably worth to move it to HTTP::Client though to make it more obvious. Or not (it belongs to Request I guess).

[sfisher]

The easy solution to this problem is to either leave off the user-agent matching requirement for your webmock or else replace it with a regular expression matcher that matches all versions of http.rb user agents.

I had this problem before and it was because I was a lazy programmer who just copied and pasted the suggested webmock when an rspec test failed. You can make webmocks as specific or broad as you want and it's somewhat well documented online or in the github Readme. It seems like people are going to a lot of work to put a specific version into their tests when there is no need to. If you don't care that http.rb is a specific version in your tests then leave the user-agent condition out or make it more broad with a regular expression.