Testing: Add failure state in testing context

[Sgitario]

Sometimes it's necessary to know whether the tests failed (at test case level or even in the before stages). For example: if we use one resource via the @QuarkusTestResource annotation that is starting a service and we don't want to stop this service when the test fails.

At the moment, we are not propagating when a test fails.

This pull request addresses this issue by adding this information in:

• QuarkusTestContext (for QuarkusTest*Callbacks)
• Context (for QuarkusTestResources)

To be precise, I've added the following methods in the above contexts:

/**
         * Get the test failure result if the test failed. When no test have failed, it will return null.
         *
         * @return the failure result that is thrown during either `BeforeAll`, `BeforeEach`, test method, `AfterAll` or
         *         `AfterEach` phases. 
         */
        TestStatus getTestStatus();

In the future, we could add even more information like the number of tests that have failed and/or the test profiles.

Fix https://github.com/quarkusio/quarkus/issues/25749

[geoand]

Hm...

I understand the logic of this, but don't these only work if called after the tests have finished?

[Sgitario]

Hm...

I understand the logic of this, but don't these only work if called after the tests have finished?

Correct. When using the after each or after all callbacks, the usage is natural. In the test resource, you would need to keep the context instance and check it in the stop method.

[geoand]

This should be very very clear in the javadoc

[geoand]

I am also wondering whether the calls should be collapsed into a single call (properly named to indicate it's intended use) and returning an object containing the two pieces of information

[Sgitario]

I am also wondering whether the calls should be collapsed into a single call (properly named to indicate it's intended use) and returning an object containing the two pieces of information

What do you mean by the calls? Perhaps, do you mean to create a custom callback QuarkusTestOnFailureCallback with the throwable instance? If so, this is something I thought of adding this callback type as well. So, let me know if you want me to add this new callback type plus update the javadocs in the Context class (for test resources).

[geoand]

Sorry, I meant adding one method instead of two, where that method returns an object that contains the necessary information.

[Sgitario]

Sorry, I meant adding one method instead of two, where that method returns an object that contains the necessary information.

I've updated this pull request by adding a single class plus updating the javadocs for the quarkus test resource method.

[Sgitario]

/cc @metacosm

[quarkus-bot[bot]]

---

:waning_crescent_moon: This workflow status is outdated as a new workflow run has been triggered.

---

Failing Jobs - Building c3c3fb5ee94d0fdb9d05c32ba8db445204c1b708

Status	Name	Step	Failures	Logs	Raw logs
:heavy_check_mark:	JVM Tests - JDK 11
:hourglass:	JVM Tests - JDK 11 Windows	Build	:warning: Check →	Logs	Raw logs
:heavy_check_mark:	JVM Tests - JDK 17

[quarkus-bot[bot]]

---

:waning_crescent_moon: This workflow status is outdated as a new workflow run has been triggered.

---

Failing Jobs - Building 630d3defdf3d99985936f38f47e9902b61cb332e

Status	Name	Step	Failures	Logs	Raw logs
✖	JVM Tests - JDK 11	Build	Failures	Logs	Raw logs
:heavy_check_mark:	JVM Tests - JDK 17

Full information is available in the Build summary check run.

Failures

:gear: JVM Tests - JDK 11 #

- Failing: extensions/smallrye-reactive-messaging-kafka/deployment 
! Skipped: integration-tests/kafka-oauth-keycloak integration-tests/kafka-sasl-elytron integration-tests/kubernetes/quarkus-standard-way-kafka and 3 more

:package: extensions/smallrye-reactive-messaging-kafka/deployment

✖ io.quarkus.smallrye.reactivemessaging.kafka.deployment.testing.KafkaDevServicesContinuousTestingTestCase.testContinuousTestingScenario1 line 59 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Failed to wait for test run 4 State{lastRun=4, running=true, inProgress=false, run=1, passed=0, failed=1, skipped=0, isBrokenOnly=false, isTestOutput=false, isInstrumentationBasedReload=false, isLiveReload=true}
	at io.quarkus.test.ContinuousTestingTestUtils.waitForNextCompletion(ContinuousTestingTestUtils.java:44)
	at io.quarkus.smallrye.reactivemessaging.kafka.deployment.testing.KafkaDevServicesContinuousTestingTestCase.testContinuousTestingScenario1(KafkaDevServicesContinuousTestingTestCase.java:59)

[quarkus-bot[bot]]

---

:waning_crescent_moon: This workflow status is outdated as a new workflow run has been triggered.

---

Failing Jobs - Building 74dbcddebbc1d7aac4baf040ab5a185ce1beb806

Status	Name	Step	Failures	Logs	Raw logs
✖	JVM Tests - JDK 11	Build	Failures	Logs	Raw logs
:heavy_check_mark:	JVM Tests - JDK 17

Full information is available in the Build summary check run.

Failures

:gear: JVM Tests - JDK 11 #

- Failing: integration-tests/grpc-hibernate 

:package: integration-tests/grpc-hibernate

✖ com.example.grpc.hibernate.BlockingRawTest.shouldAdd line 61 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in com.example.grpc.hibernate.BlockingRawTest that uses java.util.List was not fulfilled within 30 seconds.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)

[quarkus-bot[bot]]

---

:waning_crescent_moon: This workflow status is outdated as a new workflow run has been triggered.

---

Failing Jobs - Building 3f7d62d5545bc90d9bf4bb69e595633465b13eab

Status	Name	Step	Failures	Logs	Raw logs
:heavy_check_mark:	JVM Tests - JDK 11
✖	JVM Tests - JDK 17	Build	Failures	Logs	Raw logs

Full information is available in the Build summary check run.

Failures

:gear: JVM Tests - JDK 17 #

- Failing: integration-tests/elytron-undertow integration-tests/grpc-hibernate 

:package: integration-tests/elytron-undertow

✖ io.quarkus.it.undertow.elytron.BaseAuthTest.testPost line 27 - More details - Source on GitHub

java.lang.AssertionError: 
1 expectation failed.
Expected status code <200> but was <500>.

:package: integration-tests/grpc-hibernate

✖ com.example.grpc.hibernate.BlockingRawTest.shouldAdd line 61 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Condition with com.example.grpc.hibernate.BlockingRawTest was not fulfilled within 30 seconds.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)

[Sgitario]

PR updated with conflicts resolved. cc @metacosm @gsmet

[gsmet]

I like it better but I prefer if we get @geoand to review and merge it. Let's wait for him to come back from the dead :).

[geoand]

I likely won't be back this week

[gsmet]

@metacosm you around to test that one?

[metacosm]

@gsmet I need to check with the new changes, I was OK with the initial version.

[metacosm]

As long as the test result is available from the context so that a test resource can figure out if tests have failed at some point, that's good enough for me 😄

[geoand]

So should we merge this one?

[Sgitario]

So should we merge this one?

Yes, please. I've just rebased the pull request just in case.

[geoand]

Great, thanks!

[quarkus-bot[bot]]

---

:waning_crescent_moon: This workflow status is outdated as a new workflow run has been triggered.

---

Failing Jobs - Building 0302edc043e06eb98da993c59c252d77dc576f16

Status	Name	Step	Failures	Logs	Raw logs
✖	Initial JDK 11 Build	Build	Failures	Logs	Raw logs

Failures

:gear: Initial JDK 11 Build #

- Failing: test-framework/junit5 
! Skipped: devtools/bom-descriptor-json devtools/cli docs and 394 more

:package: test-framework/junit5

✖ Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project quarkus-junit5: Compilation failure

[quarkus-bot[bot]]

---

:waning_crescent_moon: This workflow status is outdated as a new workflow run has been triggered.

---

Failing Jobs - Building 8ccb59bb98c55d4b7098f4f2a4a1a13df150a98e

Status	Name	Step	Failures	Logs	Raw logs
:heavy_check_mark:	JVM Tests - JDK 11
:heavy_check_mark:	JVM Tests - JDK 17
✖	JVM Tests - JDK 18	Build	Failures	Logs	Raw logs
✖	MicroProfile TCKs Tests	Verify	Failures	Logs	Raw logs

Full information is available in the Build summary check run.

Failures

:gear: JVM Tests - JDK 18 #

- Failing: integration-tests/oidc-client-reactive 

:package: integration-tests/oidc-client-reactive

✖ io.quarkus.it.keycloak.OidcClientTest.testGetUserNameReactive - More details - Source on GitHub

java.lang.RuntimeException: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.test.junit.QuarkusTestExtension.throwBootFailureException(QuarkusTestExtension.java:701)
	at io.quarkus.test.junit.QuarkusTestExtension.interceptTestClassConstructor(QuarkusTestExtension.java:772)

---

:gear: MicroProfile TCKs Tests #

- Failing: tcks/resteasy-reactive 

:package: tcks/resteasy-reactive

✖ Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.0.0:exec (test) on project quarkus-tck-resteasy-reactive: Command execution failed.

:package: tcks/resteasy-reactive/target/testsuite/tests

✖ com.sun.ts.tests.jaxrs.ee.rs.core.configurable.JAXRSClient0133.registerClassNotAssignableContractsInMapTest line 924 - More details - Source on GitHub

com.sun.ts.tests.jaxrs.common.JAXRSCommonClient$Fault: unexpected number of registered classes found: 1 on Client configuration
	at com.sun.ts.tests.jaxrs.ee.rs.core.configurable.JAXRSClient0133.checkConfig(JAXRSClient0133.java:1227)
	at com.sun.ts.tests.jaxrs.ee.rs.core.configurable.JAXRSClient0133.registerClassNotAssignableContractsInMapTest(JAXRSClient0133.java:924)

✖ com.sun.ts.tests.jaxrs.ee.rs.core.configurable.JAXRSClient0133.registerObjectNotAssignableContractsInMapTest line 1139 - More details - Source on GitHub

com.sun.ts.tests.jaxrs.common.JAXRSCommonClient$Fault: unexpected number of registered classes found: 1 on Client configuration
	at com.sun.ts.tests.jaxrs.ee.rs.core.configurable.JAXRSClient0133.checkConfig(JAXRSClient0133.java:1227)
	at com.sun.ts.tests.jaxrs.ee.rs.core.configurable.JAXRSClient0133.registerObjectNotAssignableContractsInMapTest(JAXRSClient0133.java:1139)

✖ com.sun.ts.tests.jaxrs.ee.rs.core.configurable.JAXRSClient0133.registerObjectNotAssignableContractsTest line 669 - More details - Source on GitHub

com.sun.ts.tests.jaxrs.common.JAXRSCommonClient$Fault: unexpected number of registered classes found: 1 on Client configuration
	at com.sun.ts.tests.jaxrs.ee.rs.core.configurable.JAXRSClient0133.checkConfig(JAXRSClient0133.java:1227)
	at com.sun.ts.tests.jaxrs.ee.rs.core.configurable.JAXRSClient0133.registerObjectNotAssignableContractsTest(JAXRSClient0133.java:669)

✖ com.sun.ts.tests.jaxrs.ee.rs.core.configurable.JAXRSClient0133.registerClassNotAssignableContractsTest line 294 - More details - Source on GitHub

com.sun.ts.tests.jaxrs.common.JAXRSCommonClient$Fault: unexpected number of registered classes found: 1 on Client configuration
	at com.sun.ts.tests.jaxrs.ee.rs.core.configurable.JAXRSClient0133.checkConfig(JAXRSClient0133.java:1227)
	at com.sun.ts.tests.jaxrs.ee.rs.core.configurable.JAXRSClient0133.registerClassNotAssignableContractsTest(JAXRSClient0133.java:294)

[quarkus-bot[bot]]

---

:waning_crescent_moon: This workflow status is outdated as a new workflow run has been triggered.

---

Failing Jobs - Building 05185afa72dc9810f776684cfc940cf2d81568fe

Status	Name	Step	Failures	Logs	Raw logs
:heavy_check_mark:	JVM Tests - JDK 11
✖	JVM Tests - JDK 17	Build	Failures	Logs	Raw logs
✖	JVM Tests - JDK 18	Build	Failures	Logs	Raw logs

Full information is available in the Build summary check run.

Failures

:gear: JVM Tests - JDK 17 #

- Failing: extensions/smallrye-reactive-messaging-kafka/deployment 
! Skipped: integration-tests/kafka-oauth-keycloak integration-tests/kafka-sasl-elytron integration-tests/kubernetes/quarkus-standard-way-kafka and 3 more

:package: extensions/smallrye-reactive-messaging-kafka/deployment

✖ io.quarkus.smallrye.reactivemessaging.kafka.deployment.testing.KafkaDevServicesContinuousTestingTestCase.testContinuousTestingScenario1 line 59 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Failed to wait for test run 4 State{lastRun=3, running=true, inProgress=true, run=1, passed=0, failed=1, skipped=0, isBrokenOnly=false, isTestOutput=false, isInstrumentationBasedReload=false, isLiveReload=true}
	at io.quarkus.test.ContinuousTestingTestUtils.waitForNextCompletion(ContinuousTestingTestUtils.java:44)
	at io.quarkus.smallrye.reactivemessaging.kafka.deployment.testing.KafkaDevServicesContinuousTestingTestCase.testContinuousTestingScenario1(KafkaDevServicesContinuousTestingTestCase.java:59)

---

:gear: JVM Tests - JDK 18 #

- Failing: extensions/jdbc/jdbc-db2/deployment integration-tests/narayana-lra 
! Skipped: integration-tests/jpa-db2 

:package: extensions/jdbc/jdbc-db2/deployment

✖ Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M7:test (default-test) on project quarkus-jdbc-db2-deployment: There was a timeout in the fork

:package: integration-tests/narayana-lra

✖ org.acme.quickstart.lra.LRAParticipantTest.testLRA - More details - Source on GitHub

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at io.quarkus.test.junit.QuarkusTestExtension.throwBootFailureException(QuarkusTestExtension.java:701)
	at io.quarkus.test.junit.QuarkusTestExtension.interceptTestClassConstructor(QuarkusTestExtension.java:772)

[quarkus-bot[bot]]

Failing Jobs - Building 7f3bfb2b8cd76f5010fadc469bf8b18208a9c3c0

Status	Name	Step	Failures	Logs	Raw logs
:heavy_check_mark:	Gradle Tests - JDK 11
✖	Gradle Tests - JDK 11 Windows	Build	Failures	Logs	Raw logs
✖	JVM Tests - JDK 11	Build	Failures	Logs	Raw logs
:heavy_check_mark:	JVM Tests - JDK 17
✖	JVM Tests - JDK 18	Build	Failures	Logs	Raw logs

Full information is available in the Build summary check run.

Failures

:gear: Gradle Tests - JDK 11 Windows #

- Failing: integration-tests/gradle 

:package: integration-tests/gradle

✖ io.quarkus.gradle.devmode.CompositeBuildWithDependenciesDevModeTest.main line 24 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in io.quarkus.test.devmode.util.DevModeTestUtils that uses java.util.function.Supplier, java.util.function.Supplierjava.util.concurrent.atomic.AtomicReference, java.util.concurrent.atomic.AtomicReferencejava.lang.String, java.lang.Stringboolean was not fulfilled within 1 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)

---

:gear: JVM Tests - JDK 11 #

- Failing: integration-tests/micrometer-prometheus 

:package: integration-tests/micrometer-prometheus

✖ io.quarkus.it.micrometer.prometheus.ClientRequestTest.testClientRequests line 36 - More details - Source on GitHub

java.lang.AssertionError: 
1 expectation failed.
Response body doesn't match expectation.

---

:gear: JVM Tests - JDK 18 #

- Failing: extensions/smallrye-reactive-messaging-kafka/deployment 
! Skipped: integration-tests/kafka-oauth-keycloak integration-tests/kafka-sasl-elytron integration-tests/kubernetes/quarkus-standard-way-kafka and 3 more

:package: extensions/smallrye-reactive-messaging-kafka/deployment

✖ io.quarkus.smallrye.reactivemessaging.kafka.deployment.testing.KafkaDevServicesContinuousTestingTestCase.testContinuousTestingScenario2 line 83 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Failed to wait for test run 4 State{lastRun=3, running=true, inProgress=true, run=1, passed=0, failed=1, skipped=0, isBrokenOnly=false, isTestOutput=false, isInstrumentationBasedReload=false, isLiveReload=true}
	at io.quarkus.test.ContinuousTestingTestUtils.waitForNextCompletion(ContinuousTestingTestUtils.java:44)
	at io.quarkus.smallrye.reactivemessaging.kafka.deployment.testing.KafkaDevServicesContinuousTestingTestCase.testContinuousTestingScenario2(KafkaDevServicesContinuousTestingTestCase.java:83)

[gsmet]

@geoand you marked this one as waiting for ci so I suppose it was good on your side? CI failures are just flakes.