webmock Live requests being made when running specs (> v3.1.1)

Context

A Rails API mode app that stubs requests to a dummy Sinatra API, and utilizing fixtures for request and response bodies.

The issue occurred when upgrading Rails 5.1.5 -> 5.2.0 Ruby version is 2.5.1 WebMock version prior to upgrade was 3.0.1 WebMock upgraded to 3.4.2

WebMock.disable_net_connect!(allow_localhost: true) seems to be ignored

The following code in spec_helper.rb was working prior to the upgrade, but now live requests are being made to an external API:

require 'webmock/rspec'
require 'fantaskspec'
require 'support/bot_settings'

WebMock.disable_net_connect!(allow_localhost: true)

RSpec.configure do |config|
  config.expect_with :rspec do |expectations|
    expectations.include_chain_clauses_in_custom_matcher_descriptions = true
  end

  # request faking
  config.before(:each) do
    stub_request(:any, /api.pro.coinbase.com/).to_rack(FakeExchange)
  end

  config.mock_with :rspec do |mocks|
    mocks.verify_partial_doubles = true
  end
  config.shared_context_metadata_behavior = :apply_to_host_groups

  config.disable_monkey_patching!

  if config.files_to_run.one?
    config.default_formatter = 'doc'
  end

  config.profile_examples = 10
  config.order = :random
end

Here is FakeExchange, which was being called prior to upgrade and is unchanged:

# frozen_string_literal: true

require 'sinatra/base'

class FakeExchange < Sinatra::Base
  get "/products/#{ENV['PRODUCT_ID']}/book" do
    return json_response 200, 'depth.json' if params[:level]
    json_response 200, 'quote.json'
  end

  get "/orders" do
    json_response 200, 'open_orders.json'
  end

  get "/accounts" do
    json_response 200, 'funds.json'
  end

  delete "/orders/:id" do
    json_response 200, 'cancel_order.json'
  end

  get "/orders/:id" do
    json_response 200, "order_#{params[:id]}.json"
  end

  post "/orders" do
    params = JSON.parse(request.body.read)
    json_response 200, "#{params['side']}_order.json"
  end

  get "/fills" do
    json_response 200, "fill_#{params[:order_id]}.json"
  end

  private

  def json_response(response_status, file_name)
    content_type :json
    status response_status
    File.read("#{::Rails.root}/spec/fixtures/files/#{file_name}")
  end
end

I’ve tracked down the WebMock version change where the issue occurs - it’s any version after 3.1.1 Made me think it’s perhaps related to issue #767?

Here’s a partial RSpec trace using --format documentation where a live request to the real exchange is seen (WebMock version 3.2.0 is being used here):

Trader
  .maybe_idle
    the upcoming buy order is affordable
      does not idle
    the upcoming buy order is unaffordable
      idles
  .monitor_scrum
"e =============================="
#<Coinbase::Exchange::BadRequestError: {"message":"Invalid order id"}>
"e =============================="
#<Coinbase::Exchange::BadRequestError: {"message":"Invalid order id"}>

Live requests are being made to the exchange as indicated by Coinbase::Exchange::BadRequestError. The message {"message":"Invalid order id"} is being returned from the live exchange b/c the fixture files do not contain real order ids.

NOTE: The live requests only occur when running the entire spec suite, and not when running individual tests and files. Maybe there is some new configuration or syntax required that I am unaware of?

Dec 01 '18 18:12 TimothyClayton

@TimothyClayton have you found the reason so far? If not, would you be able to find the latest commit from WebMock master, that works?

3.1.1 is commit 4caabce21a803a33352e9434d34785323abdca82 Does it work?

If it does, does any of these break your spec?

d9af3a54d73c7de1cc16cf19e05b2522c4f7d093 08b41bcbeb913617ee1d4dcb128fb077449da7f7

Dec 27 '18 01:12 bblimke

@bblimke We got the same issue. We experienced from version 2.0.0. It is weird because an individual test is working as we expected, but when we are trying to run the whole tests the first is ok, webmock stub the request and next time it nothing to do, and the test hit the real server.