Wipes fail but self-test still says all tests are successful

[RobertSzkutak]

Wipes fail but self-test still says all tests are successful

Short description of the problem:

Sample of my Roxy log:

I, [2017-08-04T08:03:55.319564 #18192]  INFO -- : Wiping MarkLogic setup for your project from localhost...
D, [2017-08-04T08:03:55.319696 #18192] DEBUG -- : [POST]        http://localhost:8000/v1/eval
D, [2017-08-04T08:03:55.496324 #18192] DEBUG -- : code: 200
D, [2017-08-04T08:03:55.496405 #18192] DEBUG -- : XDMP-AS: (err:XPTY0004) $properties as map:map -- Invalid coercion: fn:true() as map:map
See MarkLogic Server error log for more details.
E, [2017-08-04T08:03:55.496433 #18192] ERROR -- : XDMP-AS: (err:XPTY0004) $properties as map:map -- Invalid coercion: fn:true() as map:map
See MarkLogic Server error log for more details.
E, [2017-08-04T08:03:55.496443 #18192] ERROR -- : ... Wipe FAILED
...........

Finished tests in 169.816727s, 0.1001 tests/s, 0.2944 assertions/s.

17 tests, 50 assertions, 0 failures, 0 errors, 0 skips

What are the steps to reproduce the problem?

1. Break wiping with a run-time error
2. Run wipe
3. Observe that self-test does not detect the wipe failure as a test failure

[grtjn]

Odd. Can't seem to reproduce. I broke call to setup:do-wipe, and I end up with an HTTP 500 which causes an Exception to get raised:

I, [2017-09-05T15:38:24.777616 #37664]  INFO -- : Wiping self-test deployment..
D, [2017-09-05T15:38:24.777888 #37664] DEBUG -- : (##none##), (##none##), (##none##)
I, [2017-09-05T15:38:24.777912 #37664]  INFO -- : Wiping MarkLogic setup for your project from ml9-ml1...
D, [2017-09-05T15:38:24.779128 #37664] DEBUG -- : [POST]	http://ml9-ml1:8000/v1/eval
E..........

Finished tests in 174.230532s, 0.0976 tests/s, 0.2870 assertions/s.

  1) Error:
test_0001_should bootstrap successfully twice consecutively(ServerConfig::bootstrap):
Net::HTTPFatalError: 500 "Internal Server Error"
    /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http/response.rb:119:in `error!'
    /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/http/response.rb:128:in `value'
    /Users/gjosten/Projects/github-grtjn/roxy/deploy/lib/MLClient.rb:114:in `go'
    /Users/gjosten/Projects/github-grtjn/roxy/deploy/lib/server_config.rb:2709:in `execute_query_8'
    /Users/gjosten/Projects/github-grtjn/roxy/deploy/lib/server_config.rb:564:in `execute_query'
    /Users/gjosten/Projects/github-grtjn/roxy/deploy/lib/server_config.rb:1118:in `wipe'
    /Users/gjosten/Projects/github-grtjn/roxy/deploy/test/test_server_config.rb:104:in `block (3 levels) in <top (required)>'
    /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/minitest/spec.rb:181:in `instance_eval'
    /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/minitest/spec.rb:181:in `block in after'
...

Wondering if it makes a difference against which version of ML you run self-test. CI runs against ml7, 8 and 9..

[grtjn]

@RobertSzkutak, do you think this still needs attention?

[RobertSzkutak]

I had tested on 8. To be clear, in order to reproduce this you have to manually insert a run-time error into the XQuery wipe function such as an invalid coercion error. When that occurs, I had noticed that our testing scripts don't recognize the failure and record it at the end of the log. IIRC, I overlooked this and pushed in some bad code once as a result. I'm not sure if this also happens in other functions like bootstrap.

As far as whether it "needs" to be fixed or not is probably up to us as we're the only ones who rely on self-test to ensure our code isn't broken. I don't personally have an issue with sidelining it for now.

[grtjn]

I had a closer look at the test script this morning, and discovered some oddities. And yes, wipe is executed in the after step, which is not monitored. I changed the test code in one of the last PRs, which also moved the wipe inside the assertions section! :)