noconnor
Total JUnit Tests Count in JUnitPerf context
when I run my unit tests without JUnitPerf suite
the total count is 129 excluding ignored ones
When I include JUnitPerf suite with totalExecutions=1
Junit dashboard shows
Junit perf dashboard shows 134 methods
When I include JUnitPerf suite with totalExecutions=2
134
TestExAdd : testAddition
TestCounter : testToStringWhenCounterIsZero
TestCounter : testIncrement
TestExAdd : testAddition
TestCustomHCMatcher : assertThatIsMultipleOf3
TestMultiScrubber : testRandom
TestAssertJ : testAssertions
TestCounterThreadSafety : testThreadSafety
TestBBRandom : testFalseThenTrueBB
TestBBRandom : testRand
TestBBRandom : testTrueThenFalseBB
TestBAWStub : testWithdraw
TestBAWMock : testWithdrawWithSufficientBalanceAndAccess
TestBAWMock : testWithdrawWithInsufficientBalance
TestBAWMock : testWithdrawWithoutAccess
TestBAWMock : testWithdrawalWithInsufficientFunds
TestApproval : testStringArray
TestApproval : testHashMap
TestBAWDummy : testDeposit
TestProduct : testHasTotalPrice
TestAsyncEx : testCatchUncaughtExceptions
TestPerson : testIsAdult
TestDateScrubber : testDate
TestExSub : testSubtraction
TestHamcrest : assertThatIsSame
TestHamcrest : assertThatBothAnd
TestHamcrest : assertThatMatchesRegex
TestHamcrest : assertThatIsInstanceOf
TestHamcrest : assertThatContainsString
TestHamcrest : assertThatLessThan
TestHamcrest : assertThatGreaterThan
TestHamcrest : assertThatIsNull
TestHamcrest : assertThatComparesTo
TestHamcrest : assertThatIsNotNull
TestHamcrest : assertThatIs
TestHamcrest : assertThatGreaterThanOrEqualTo
TestHamcrest : assertThatEitherOr
TestHamcrest : testHasItem
TestHamcrest : testCloseTo
TestHamcrest : assertThatStartsWith
TestHamcrest : assertThatAllOf
TestHamcrest : testHasProperty
TestHamcrest : assertThatIsEmpty
TestHamcrest : testHasItems
TestHamcrest : assertThatLessThanOrEqualTo
TestHamcrest : assertThatAnyOf
TestHamcrest : assertThatEndsWith
TestHamcrest : assertThatNot
TestGUIDScrubber : testGUIDs
TestJU5CustomAssertions : testWithCustomAssertion
TestExGroupAsserts : testGroupAssertsExecOrder
TestBAWFake : testWithdrawWithInsufficientFunds
TestBAWFake : testWithdrawWithAuthenticationFailure
TestBAWFake : testGetBalance
TestBAWFake : testWithdrawWithSufficientFunds
TestBAWFake : testDeposit
TestStringHelper : testNormalize
TestBBOrdered : testUnbalancedBraces
TestBBOrdered : testBalancedBraces
TestAssumptions : testAssumeTrueOnTestEnvWithFalseConditionWithMessageSupplier (skipped)
TestAssumptions : testAssumingThat
TestAssumptions : testAssumeFalseWithFalseCondition
TestAssumptions : testAssumeTrueOnTestEnv
TestAssumptions : testAssumeFalseWithTrueConditionWithMessageSupplier (skipped)
TestRegXScrubber : testRandom
TestExAdd : testAddition
TestBBOrdered : testUnbalancedBraces
TestBBOrdered : testBalancedBraces
TestBBRandom : testFalseThenTrueBB
TestBBRandom : testRand
TestBBRandom : testTrueThenFalseBB
TestBBOrdered : testUnbalancedBraces
TestBBOrdered : testBalancedBraces
TestBBRandom : testFalseThenTrueBB
TestBBRandom : testRand
TestBBRandom : testTrueThenFalseBB
TestCounter : testIncrement
TestCounter : testToStringWhenCounterIsZero
TestExAdd : testAddition
TestCustomHCMatcher : assertThatIsMultipleOf3
TestAssertJ : testAssertions
TestCounterThreadSafety : testThreadSafety
TestBBRandom : testFalseThenTrueBB
TestBBRandom : testRand
TestBBRandom : testTrueThenFalseBB
TestBAWStub : testWithdraw
TestBAWMock : testWithdrawWithoutAccess
TestBAWMock : testWithdrawWithSufficientBalanceAndAccess
TestBAWMock : testWithdrawalWithInsufficientFunds
TestBAWMock : testWithdrawWithInsufficientBalance
TestBAWDummy : testDeposit
TestProduct : testHasTotalPrice
TestAsyncEx : testCatchUncaughtExceptions
TestPerson : testIsAdult
TestExSub : testSubtraction
TestHamcrest : assertThatIsNull
TestHamcrest : assertThatIsSame
TestHamcrest : assertThatLessThan
TestHamcrest : assertThatMatchesRegex
TestHamcrest : assertThatContainsString
TestHamcrest : assertThatGreaterThan
TestHamcrest : assertThatIsInstanceOf
TestHamcrest : assertThatBothAnd
TestHamcrest : testHasItem
TestHamcrest : assertThatStartsWith
TestHamcrest : assertThatGreaterThanOrEqualTo
TestHamcrest : assertThatEitherOr
TestHamcrest : assertThatIs
TestHamcrest : assertThatLessThanOrEqualTo
TestHamcrest : assertThatIsNotNull
TestHamcrest : assertThatComparesTo
TestHamcrest : assertThatNot
TestHamcrest : testCloseTo
TestHamcrest : assertThatAnyOf
TestHamcrest : testHasProperty
TestHamcrest : testHasItems
TestHamcrest : assertThatEndsWith
TestHamcrest : assertThatIsEmpty
TestHamcrest : assertThatAllOf
TestJU5CustomAssertions : testWithCustomAssertion
TestExGroupAsserts : testGroupAssertsExecOrder
TestBAWFake : testGetBalance
TestBAWFake : testWithdrawWithInsufficientFunds
TestBAWFake : testDeposit
TestBAWFake : testWithdrawWithSufficientFunds
TestBAWFake : testWithdrawWithAuthenticationFailure
TestStringHelper : testNormalize
TestBBOrdered : testUnbalancedBraces
TestBBOrdered : testBalancedBraces
TestAssumptions : testAssumeFalseWithFalseCondition
TestAssumptions : testAssumeTrueOnTestEnvWithFalseConditionWithMessageSupplier (skipped)
TestAssumptions : testAssumeFalseWithTrueConditionWithMessageSupplier (skipped)
TestAssumptions : testAssumingThat
TestAssumptions : testAssumeTrueOnTestEnv
when I say totalExecutions=2, should I not expect JUnit Dashboard to show total unit tests run as double the count of what is shown for totalExecutions=1?
I think whats happing is you are running all your tests and then running a suite class annotated with @JUnitPerf that re-runs all your tests , so you run all the test twice when JUnitPerf is enabled (ie. 186 x 2 = 372)
when I say totalExecutions=2, should I not expect JUnit Dashboard to show total unit tests run as double the count of what is shown for totalExecutions=1?
No, even though the junitperf framework will run your test multiple times in a loop, to the junit test framerwork thats just 1 execution of your tests.
I think you should have 186 results in the junitperf report but i can see you only have 134.
I will look into that, i think some test names are clashing and the html reporter needs to be updated, so the test are running but they're just not being shown properly in the report.
No, even though the junitperf framework will run your test multiple times in a loop, to the junit test framerwork thats just 1 execution of your tests.
and
So the junitperf framework does not discover the test, the junit-platform-suite-engine does test discovery. The junit-platform-suite-engine test driver then iterates over each test in the suite and for each test the junitperf interceptor is >called.
Info from other issue are conflicting. May be it is better to be uniform i.e. not manipulating normal junit.. that way it is consistent..
Also in another issue you said beforeach was specially handled by JUnitPerf.. if running the tests was left to normal junit engines.. all these special cases could have been avoided. If there is a way, wish to leave tests running to JUNit Engine that way my dashboard for Junit 5 would count each of them the way it should.. b.t.w. I hardly know the internals of your implementation hence take this view as just a suggestion.
I think you should have 186 results in the junitperf report but i can see you only have 134. i think some test names are clashing and the html reporter needs to be updated,
Yes, I have seen some assert exceptions coming from html report class.. next time they come, I shall post the exact error..
I think whats happing is you are running all your tests and then running a suite class annotated with @JUnitPerf that re-runs all your tests , so you run all the test twice when
JUnitPerfis enabled (ie. 186 x 2 = 372)
👌 True.
Info from other issue are conflicting. May be it is better to be uniform i.e. not manipulating normal junit.. that way it is consistent..
Its not manipulating junit, the junitperf framework hooks into the junit InvocationInterceptor class and is called by junit when the test method is supposed to run (docs are here).
Junit will only call this method once per test.
In order to call the test method multiple times to generate latency distributions, when the InvocationInterceptor::interceptTestMethod is triggered by junit, the test method is run in a loop by the junitperf framework.
That is the only way to run a method multiple times.
Also in another issue you said beforeach was specially handled by junitperf.
Because the test method is being called multiple times, the junitperf framework needs to call before/after each explicitly to allow test state to be reset/cleared up between invocations.
This is the same behaviour a normal junit run would execute ie.
- normal run: beforeEach -> method -> afterEach -> next test
- junitperf run: loop (beforeEach -> method -> afterEach) -> next test
The latency distributions are generated by capturing the method execution time for each loop iteration.
Looks like these tests are missing from the report:
*** com.tejasoft.edu.bank.tests.doubles.spys.TestBAWSpy 2 0 0 1.856s 100%
*** com.tejasoft.edu.calc.date.tests.bdd.ccbr.suites.TestSuiteCCBRDateCalc 8 0 0 0.049s 100%
*** com.tejasoft.edu.calc.rpn.tests.bdd.ccbr.suites.TestSuiteCCBRRPNCalc 14 0 0 0.972s 100%
*** com.tejasoft.edu.dsa.bbraces.tests.ju.ju5.ut.propt.jqwik.TestPropJQwikBB 4 0 0 0.894s 100%
*** com.tejasoft.tests.ju.ju3.ut.learn.TestJU3 2 0 0 0.003s 100%
*** com.tejasoft.tests.ju.ju4.ut.learn.TestJU4 2 0 0 0.003s 100%
*** com.tejasoft.tests.ju.ju5.ut.asserts.awaitility.learn.async.tests.TestAsyncWorker 110 0 110 0.007s -
*** com.tejasoft.tests.tng.learn.TestBasic 8 0 0 0.019s 100%
*** com.tejasoft.tests.tng.learn.TestSoftAssert 6 0 0 0.023s 100%
*** com.tejasoft.utils.tests.ju.ju4.ut.pbt.jqt.TestJUQCounter 2 0 0 0.226s 100%
*** com.tejasoft.utils.tests.ju.ju5.ut.parms.TestCounter0CSVFileSource 10 0 0 0.016s 100%
*** com.tejasoft.utils.tests.ju.ju5.ut.parms.TestCounter1CSVSource 10 0 0 0.019s 100%
*** com.tejasoft.utils.tests.ju.ju5.ut.parms.TestCounter2ValueSource 16 0 0 0.013s 100%
*** com.tejasoft.utils.tests.ju.ju5.ut.parms.TestCounter3NullSource 2 0 0 0.001s 100%
*** com.tejasoft.utils.tests.ju.ju5.ut.parms.TestCounter4EmptySource 2 0 0 0.001s 100%
*** com.tejasoft.utils.tests.ju.ju5.ut.parms.TestCounter5NullEmptySource 8 0 0 0.008s 100%
*** com.tejasoft.utils.tests.ju.ju5.ut.parms.TestCounter6EnumSource 8 0 0 0.008s 100%
*** com.tejasoft.utils.tests.ju.ju5.ut.parms.TestCounter7EmptyEnumSource 2 0 0 0.029s 100%
*** com.tejasoft.utils.tests.ju.ju5.ut.parms.TestCounter8MethodSource 10 0 0 0.021s 100%
*** com.tejasoft.utils.tests.ju.ju5.ut.parms.TestCounter9ArgumentsSource 10 0 0 0.016s 100%
*** com.tejasoft.utils.tests.ju.ju5.ut.parms.aprt.TestCounterAprtValueSource 2 0 0 0.115s 100%
Seems like those tests are not picking up the @JUnitPerfTest annotation for some reason.
I have released a 1.35.0 with additional trace logging that might help identify what is different about these tests.
If you can run with that version when you get a chance and attach the junit report zip with full details again that would be helpful in troubleshooting.
, the test method is run in a loop by the
junitperfframework.That is the only way to run a method multiple times.
Does this mean these 4 million times running of each method shown the performance dashboard is done by JUnitPerf.. in 60 seconds which is the default durationMs, 4 million times is that possible?
Yes. Junit calls the junitperf interceptor. The interceptor calls the test as many times as it can in 60secs. Then the test is marked as completed.
Yes. Junit calls the junitperf interceptor.
JUnit calls JUnitPerf Once
The interceptor calls the test as many times as it can in 60secs.
JUnitPerf is calling this method
approx 14.5 million times in 60 seconds testAddition. Ok, this is clear now.
If you can run with that version when you get a chance and attach the junit report zip with full details again that would be helpful in troubleshooting.
with 1.35.0 version of JUnitPerf
- junitperf run: loop (beforeEach -> method -> afterEach) -> next test
JUnitPerf waits for the next test that should be triggered by junit?
junitperf run: loop (beforeEach -> method -> afterEach)
It is this run loop that applies performance test specification as annotated e.g @JUnitPerfTest(totalExecutions = 1000, rampUpPeriodMs=10000, threads = 20, warmUpMs = 0)
Looks like these tests are missing from the report:
These are the tests that have different engines
such as to run JUnit 4 and 3 part of JUnit 5 org.junit.vintage:junit-vintage-engine
BDD tests with io.cucumber:cucumber-junit-platform-engine
JUnit parameter tests with org.junit.jupiter:junit-jupiter-params
testng tests with org.junit.support:testng-engine
property based checking with com.pholser:junit-quickcheck-core
uses awaitablity lib org.awaitility:awaitility
TestBAWSpy was using JUnit 4 @Test annotation, which I have fixed now. In short, JUnitPerf is picking up pure JUnit 5 tests that are annotated with org.junit.jupiter.api.Test but unable to pick up junit 5 extended one by 3rd parties.. e.g. @RunWith(JUnitQuickcheck.class) etc.. full sample test that is suite method with @Property annotation
package com.tejasoft.utils.tests.ju.ju4.ut.pbt.jqt;
import com.pholser.junit.quickcheck.From;
import com.pholser.junit.quickcheck.Property;
import com.pholser.junit.quickcheck.generator.Fields;
import com.pholser.junit.quickcheck.runner.JUnitQuickcheck;
import com.tejasoft.utils.Counter;
import org.junit.runner.RunWith;
import static org.junit.Assert.assertEquals;
@RunWith(JUnitQuickcheck.class)
public final class TestJUQCounter
{
@Property
public final void incrementing(@From(Fields.class) final Counter aCounter)
{
int count = aCounter.getCount();
assertEquals(count + 1, aCounter.increment().getCount());
}
}
Yes, this makes sense now.
I can see from the logs that the interceptor is not being triggered for these tests.
The junitperf-junit5 library provides a junit5-extension to turn junit5 unittests into performance/load tests.
There is support for junit4 tests using the com.github.noconnor:junitperf library and junit rules (docs are here) . But suite level rules are not supported by junit4 , so to use Rules at the suite level for junit4 tests you would need to define a custom runner (see here).
Potentially I could add suite rule support to the junit4 junitperf library by creating a custom junit4 runner that could be applied to the suite class, but i'm not sure how that would interact with the other 3rd party runners you use in your tests.
There is no junitperf support for junit3.
Junit3, junit4 and junit5 have all defined different approaches for intercepting test methods, so there is no common approach that can be applied across all frameworks.