Internal server error when request payload cannot be encoded to UTF-8

[lextiz]

I am trying to stub a response for a PUT request that has a zip file as a payload. As far as I understand from the documentation this is a valid use case. However the stub server responds with 500 code and has the following stack trace in logs:

INFO  WEBrick 1.3.1
INFO  ruby 2.2.2 (2015-04-13) [x86_64-linux]
INFO  WEBrick::HTTPServer#start: pid=1 port=80
I, [2019-03-22T10:24:24.911391 #1]  INFO -- : Received request PUT /data/478ef4f8-fe03-4b91-9b3a-41164ce23fde/dataset.zip
E, [2019-03-22T10:24:24.923967 #1] ERROR -- : Error ocurred in mock service: Encoding::UndefinedConversionError - "\xEB" from ASCII-8BIT to UTF-8
E, [2019-03-22T10:24:24.924102 #1] ERROR -- : /pact/lib/vendor/ruby/2.2.0/gems/pact-mock_service-2.12.0/lib/pact/mock_service/request_handlers/interaction_replay.rb:16:in `encode'
/pact/lib/vendor/ruby/2.2.0/gems/pact-mock_service-2.12.0/lib/pact/mock_service/request_handlers/interaction_replay.rb:16:in `to_json'
/pact/lib/vendor/ruby/2.2.0/gems/pact-mock_service-2.12.0/lib/pact/mock_service/request_handlers/interaction_replay.rb:16:in `pretty_generate'
/pact/lib/vendor/ruby/2.2.0/gems/pact-mock_service-2.12.0/lib/pact/mock_service/request_handlers/interaction_replay.rb:49:in `find_response'
/pact/lib/vendor/ruby/2.2.0/gems/pact-mock_service-2.12.0/lib/pact/mock_service/request_handlers/interaction_replay.rb:41:in `respond'
/pact/lib/vendor/ruby/2.2.0/gems/pact-mock_service-2.12.0/lib/pact/mock_service/request_handlers/base_request_handler.rb:17:in `call'
/pact/lib/vendor/ruby/2.2.0/gems/rack-2.0.6/lib/rack/cascade.rb:33:in `block in call'
/pact/lib/vendor/ruby/2.2.0/gems/rack-2.0.6/lib/rack/cascade.rb:24:in `each'
/pact/lib/vendor/ruby/2.2.0/gems/rack-2.0.6/lib/rack/cascade.rb:24:in `call'
/pact/lib/vendor/ruby/2.2.0/gems/pact-mock_service-2.12.0/lib/pact/consumer/mock_service/cors_origin_header_middleware.rb:11:in `call'
/pact/lib/vendor/ruby/2.2.0/gems/pact-mock_service-2.12.0/lib/pact/consumer/mock_service/error_handler.rb:13:in `call'
/pact/lib/vendor/ruby/2.2.0/gems/pact-mock_service-2.12.0/lib/pact/mock_service/app.rb:33:in `call'
/pact/lib/vendor/ruby/2.2.0/gems/pact-mock_service-2.12.0/lib/pact/consumer/mock_service/set_location.rb:14:in `call'
/pact/lib/vendor/ruby/2.2.0/gems/rack-2.0.6/lib/rack/handler/webrick.rb:86:in `service'
/pact/lib/vendor/ruby/2.2.0/gems/webrick-1.3.1/lib/webrick/httpserver.rb:138:in `service'
/pact/lib/vendor/ruby/2.2.0/gems/webrick-1.3.1/lib/webrick/httpserver.rb:94:in `run'
/pact/lib/vendor/ruby/2.2.0/gems/webrick-1.3.1/lib/webrick/server.rb:191:in `block in start_thread'

Am I missing something or the stub server expects JSON payload for all interactions?

The assumption is wrong: the JSON object is Pact internal, but includes the actual payload.

The root cause of this specific failure is that encode method fails to encode some payload as UTF-8, is that is not meant to be interpreted as such. In my case no validations for body are needed, is that possible to bypass the failing code by having a different contract?

No, it is used everywhere.

[lextiz]

Fix proposal: wrap JSON parsing in a try/catch block, when parsing would fail than the object that is printed for debugging purposes would not be pretty printed, but the transaction would not fail. Would such a fix be accepted as PR?

[bethesque]

Hm... I'm really not sure how a zip file would work. How would it get written into the JSON file?

[lextiz]

What I am trying to do is to use pact stub server to return a response to a POST request, that has binary body. The attempt to encode a go object (that contains also the binary payload) to JSON and back comes from pretty_generate function, that is used only to beautify the debugging printouts. So the proposal is to wrap this logic in a try/catch block, so the transaction would not fail when pretty printing of the debug logs fails.

[bethesque]

I still don't think this is going to work. How do you expect the binary data to be encoded in the UTF-8 encoded json pact file?

[lextiz]

That works for me fine, because this data is not written to a JSON file. I am not running any assertions on the binary data (except that it exists), I just want pact-mock_service not to crash when it receives a request with non-UTF-8-encodable data. I do not think that running assertions on the binary data, that would require writing it to a JSON file is a good idea in general.

[YOU54F]

I've just come across this issue when creating a pact-test that involves uploading a pact file with a binary upload.

I found it also erroring at the same point as you, although I have had to use rescue Encoding::UndefinedConversionError

			begin
				JSON.pretty_generate(JSON.parse(object.to_json))
			  rescue JSON::ParserError
				object
			  rescue Encoding::UndefinedConversionError
				object
			  end

I am not sure if pact is going to work for my current use case (mutlipart/form-data) but spiking it today

[bethesque]

Released in 3.1.1

[YOU54F]

@lextiz does this solve your issue now? I need to try it out at work tomorrow

[lextiz]

@YOU54F I have tested the fix "in-place" only by replacing the ruby code in live environment and checking the API behavior, without releasing the whole library. The 3.1.1 version was not tested by me yet.