Improve Pact specification compliance spec

[tancnle]

Why

Provide better guidance on future specification upgrades.

What

Running rspec spec/pact_specification/compliance-1.0.0.rb with pact-specification repo checked out in certain version throws exception and short circuit all other tests.

# With v1
KeyError: key not found: :method
  /Users/tanle/.gem/ruby/2.2.3/gems/pact-support-0.6.0/lib/pact/consumer_contract/request.rb:15:in `fetch'
  /Users/tanle/.gem/ruby/2.2.3/gems/pact-support-0.6.0/lib/pact/consumer_contract/request.rb:15:in `from_hash'
  /Users/tanle/works/open-source/pact/spec/pact_specification/compliance-1.0.0.rb:27:in `block (7 levels) in <top (required)>'
  /Users/tanle/.gem/ruby/2.2.3/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:385:in `module_exec'
  /Users/tanle/.gem/ruby/2.2.3/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:385:in `subclass'
  /Users/tanle/.gem/ruby/2.2.3/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:258:in `block in define_example_group_method'
  /Users/tanle/works/open-source/pact/spec/pact_specification/compliance-1.0.0.rb:25:in `block (6 levels) in <top (required)>'
...

# With v2
KeyError: key not found: :headers
  /Users/tanle/.gem/ruby/2.2.3/gems/pact-support-0.6.0/lib/pact/consumer/request.rb:14:in `fetch'
  /Users/tanle/.gem/ruby/2.2.3/gems/pact-support-0.6.0/lib/pact/consumer/request.rb:14:in `from_hash'
  /Users/tanle/works/open-source/pact/spec/pact_specification/compliance-1.0.0.rb:28:in `block (7 levels) in <top (required)>'
  /Users/tanle/.gem/ruby/2.2.3/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:385:in `module_exec'
  /Users/tanle/.gem/ruby/2.2.3/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:385:in `subclass'
  /Users/tanle/.gem/ruby/2.2.3/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:258:in `block in define_example_group_method'
  /Users/tanle/works/open-source/pact/spec/pact_specification/compliance-1.0.0.rb:25:in `block (6 levels) in <top (required)>'
...

How

• Gracefully handle exception case, fail the test and continue on.
• Loop through those failed test cases and confirm whether those are legitimate non-compliance failures or incorrect specification test cases.

[bethesque]

👍

[YOU54F]

We've since created a pact-compatibility-suite with the intention that it can be validated across languages.

It's implementation will be tracked against these tickets

• https://github.com/pact-foundation/pact-ruby/issues/317
• https://github.com/pact-foundation/pact-ruby/issues/318
• https://github.com/pact-foundation/pact-ruby/issues/319

So work should be considered to deprecate this compliance suite from the project (it currently doesn't run as part of the tests)

note the file resides here

edit: I've now raised a separate ticket to track the implementation of the compat-suite so will close off this ticket, thanks!