wildfly-core [WFCORE-6135] TestSuite - NPE fastfail and safe tearDown

Fixes https://issues.redhat.com/browse/WFCORE-6135

Nov 25 '22 15:11 boris-unckel

Core -> Full Integration Build 11855 outcome was FAILURE using a merge of c8fd9cd8f505ee2dcdfe285c36d7bddcb4f2da13 Summary: Tests failed: 1 (1 new), passed: 3998, ignored: 54 Build time: 01:53:32

Failed tests

org.jboss.as.test.integration.jpa.webtxem.WebJPATestCase.testReadWrite: java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: HTTP Status 500 Response: <html><head><title>ERROR</title><style>
body {
    font-family: "Lucida Grande", "Lucida Sans Unicode", "Trebuchet MS", Helvetica, Arial, Verdana, sans-serif;
    margin: 5px;
}

.header {
    background-image: linear-gradient(bottom, rgb(153,151,153) 8%, rgb(199,199,199) 54%);
    background-image: -o-linear-gradient(bottom, rgb(153,151,153) 8%, rgb(199,199,199) 54%);
    background-image: -moz-linear-gradient(bottom, rgb(153,151,153) 8%, rgb(199,199,199) 54%);
    background-image: -webkit-linear-gradient(bottom, rgb(153,151,153) 8%, rgb(199,199,199) 54%);
    background-image: -ms-linear-gradient(bottom, rgb(153,151,153) 8%, rgb(199,199,199) 54%);
    
    background-image: -webkit-gradient(
        linear,
        left bottom,
        left top,
        color-stop(0.08, rgb(153,151,153)),
        color-stop(0.54, rgb(199,199,199))
    );
    color: black;
    padding: 2px;
    font-weight: normal;
    border: solid 1px;
    font-size: 170%;
    text-align: left;
    vertical-align: middle; 
    height: 32px; 
    margin-bottom: 10px;
}
.error-div {
    display: inline-block;
    width: 32px;
    height: 32px;
    background: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QAAAAAAAD5Q7t/AAAACXBIWXMAAABIAAAASABGyWs+AAAACXZwQWcAAAAgAAAAIACH+pydAAAGGElEQVRYw8WXW2wcVxnHf+fM7NXrdXbdtZ3ipGqCEzvQKmlFoUggwkvvGBoaqVVV5Y0nkCoQPPCAhMQbQlzUh/IWHoh6QUVKZVLxUCGogKZRW9uJkyg0iVzZ2dhre9c7u7M7c87hYWc2M7t2bBASI306O/85+v7/7zLn24H/8yV2u/EcFNvwVNK2X0DKw1qpYaX1gCWlY0m5arRe8JQ6I2DmaVj/nwk4CxO2Zf1aw9dKo6N67+eOZHN7x0gViiSHBmlXa7hrazjLt1i+eKmxUi5bwpg/e8a8PA3X/msBr0MyJ+WvgFOffehY8v7HH5OW78HqGjgOtNrge2AnIJWAgQEYLqBsm3/NvKM/mZ1rYczpRa2/9x3w/iMBM1BCiHfuGRmZevClF9NJ5cP1m9B0+/aa3t/ZDNw3TtuSfPy737trK6sXtTGPPwOruxLwNhSEEHMHpqZGDz93wubKFdio3ZXUGNOHs2cIDh/g6pt/9G8uXLnlGPP5k1Dt5bN60m6lhXj34OTkxKET37SZm4d6oz/KgFRH7yOrBozbwlTWGf7ql6RZrWTcytrxaTj9Ro9OGb3JS/mz0nDxgYnnvmWbuUsdJ6FTY2JmImShqZA83N90UbOXOXDiG4nCcPFoTsqfbluCP8DenJQLx1/+7pBcXILaZl9qo2svFmalb48xkM9hxkf5+29+u9HSenIayn0ZyEr5i0NHpnKy7aGrtW6UehsLozRBRlRQEh3BwzLpWh3h++yfPJRLSvnzvhK8DkPamOl7n3nCUjcXtyXuK0X0WViaAFc9QtSNJe594uu2MubZGcjHBGTgqeFiQQml0K12nDgSpd4iyi4uRMci6Q8xJSWq7SGEZCg/qD14LCYgIcTzoxMHc6qygYk40aETKTFSdh2aANOWhbEsjJSYCK6C/V0LcH+9xvCB/TlbiBdCAXbQMJPZ/fvQjUbXEULAa6/1deyuh0dwNU+e7DSzEOhmi8xn9iI+ujgVE6ChlBwpYcqVjuIgA3IXBMYYtNbd30IIpJQI0ZEa+gPQnk9ieA8GSrESaGNyiWIBrVQntUHqdrq01iiluo0WilBK4fs+WutOr4S9YAyJoRzamMGYAClEve046ETyTr13EKCU6kZ+N4GNCxfwKpVOryQSuOUVhBCbsRIIuN1cWRkaSCfRrdaOTncijgltNPDn5xHZLIn7x3GNh4CVmADgcv3a9YmBY0cxm06nAQH31KlOao3BL5fxbtzAOE73hAt7JXb6RU7PGO44WK5Do1ZHw6WYAN+YM+X5S8dLXziWizVNvY5aWcFfWuq8Idsctb1Yr8DwmRwrUr1yfVMbcyYmwIWZ2mbdai9+gvfBAsaAbrUwrrtllLuJPibIGGQ2hUlIHLdlA+diTXgSqkKIt5b/OevL+8bwqlWU63anW3jyhaa4c9T2Hs0hrsL5EOCJQ/sov7/gS8Mb07AZEwDgav2D5aVy3eSziEL+Dllk0PSRcmcMqx6BUeGykINsksrttboPP4w2aVfACVhGiFevvft+I/XwJCad3DF6tc2MiD4zmRSZhyZY/OusI4V4JTqKYwIA6lr/2Gm6s5/+7UMv/egDmEyqL7VbRR/LViR6sikGvniYpffm2m6r/VFd65/0vqZ9R/tbsMeG+WKpMDL+lWMJ58Jl/NVqvOG26PAoDmCXhhg4epDl9+a9zUp1uQkPbvWfcMvZchbuEXAuk0kf2X/84YxpuDQv3kA5zW1fw9CswSyZyXFEJsmnf/m46bntuTY8+SxUtuLadri9CokxeEUK8WJhtJgaeeSIxG3RurWGv+Gg3RbaU5CwkOkU1lCWxFgBmbS4ff6qX7297rXh9Gn4/llobMcjtsFywACQ+zZMTsOP8vBIOpNS+bFiJj2cxx7MYg+k8ZwmrWqD9lqNjVvrTc9tWRtw/k345dtwFagDTmTdUUA2EDAYXffB2JPw6FH4ch7GUpCzIemD50J9A1Y+hH/8Cc4vdTq9Tud9D9dNOt+MajclyNL53xZmIhusmcBSQJLOd4UJnHqACzTppDy0kHizl/yuPRB5nggIM0A6IE4GuA34EQFtoBUQuwGm7kbwb+eaEEXmuV5dAAAAJXRFWHRjcmVhdGUtZGF0ZQAyMDA5LTExLTEwVDE5OjM4OjI0LTA3OjAwdDKp4gAAACV0RVh0ZGF0ZTpjcmVhdGUAMjAxMC0wMi0yMFQyMzoyNjoyNC0wNzowMC7DUNYAAAAldEVYdGRhdGU6bW9kaWZ5ADIwMTAtMDEtMTFUMDg6NTc6MzUtMDc6MDCruapPAAAAMnRFWHRMaWNlbnNlAGh0dHA6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvUHVibGljX2RvbWFpbj/96s8AAAAldEVYdG1vZGlmeS1kYXRlADIwMDktMTEtMTBUMTk6Mzg6MjQtMDc6MDArg9/WAAAAGXRFWHRTb3VyY2UAVGFuZ28gSWNvbiBMaWJyYXJ5VM/tggAAADp0RVh0U291cmNlX1VSTABodHRwOi8vdGFuZ28uZnJlZWRlc2t0b3Aub3JnL1RhbmdvX0ljb25fTGlicmFyebzIrdYAAAAASUVORK5CYII=') left center no-repeat;
}
.error-text-div {
    display: inline-block;
    vertical-align: top;
    height: 32px;
}
.label {
    font-weight:bold;
    display: inline-block;
}
.value {
    display: inline-block;
    margin-left: 5px;
}
pre {
    font-size: 110%;

Nov 25 '22 17:11 wildfly-ci

Core -> WildFly Preview Integration Build 11995 outcome was UNKNOWN using a merge of df928d6df13c5317534fa9593b9e89c17bff23c8 Summary: Canceled Build time: 00:01:01

Nov 26 '22 12:11 wildfly-ci

Core -> Full Integration Build 11856 outcome was UNKNOWN using a merge of df928d6df13c5317534fa9593b9e89c17bff23c8 Summary: Canceled Build time: 00:01:02

Nov 26 '22 12:11 wildfly-ci

Core -> Full Integration Build 12041 outcome was FAILURE using a merge of df928d6df13c5317534fa9593b9e89c17bff23c8 Summary: Tests failed: 1 (1 new), passed: 7296, ignored: 153 Build time: 03:28:24

Failed tests

org.wildfly.extension.undertow.UndertowSubsystem10TestCase.testRuntime: java.lang.AssertionError: expected:<2> but was:<0>
	at org.wildfly.extension.undertow.AbstractUndertowSubsystemTestCase.testRuntime(AbstractUndertowSubsystemTestCase.java:140)
	at org.wildfly.extension.undertow.UndertowSubsystem10TestCase.testRuntime(UndertowSubsystem10TestCase.java:74)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

Nov 26 '22 16:11 wildfly-ci

@yersan Thanks for the detailed review. I'm going to fix the issues tomorrow. I want to bring one thing to your attention: BeforeClass and AfterClass are independent. Regardless whether BeforeClass breaks or not, AfterClass is called. You can't assume from the broken state of BeforeClass to the beginning of AfterClass.

I agree that things which cause an error today should not be silent: This change will cause failures (as term of JUnit) then.

Dec 21 '22 20:12 boris-unckel

In such a hypothetical case, we should fail, however by applying these changes we no longer fail, we will silently continue without checking whether the cleanup routine is executed, making the test weaker.

Yes, agreed: A test failing today with a late error shall fail with an early failure after the change. The test keeps strong, respecting Before/After behaviour and making implicit assumptions explicit.

Could you also describe a bit more how AdminOnlyModeTestCase would help you with the https://issues.redhat.com/browse/WFCORE-6135 description?

I have updated the description, adding the current situation, describing the JUnit behaviour. I have changed the solution proposal, to respect the discussed issues.

Dec 22 '22 08:12 boris-unckel

@boris-unckel , thanks!

Jan 17 '23 17:01 yersan