jqwik-team
Add debugging mode so arbitraries are easier to understand in the Java debugger
Testing Problem
Understanding the nature of Arbitrary, Combinator, etc in the runtime debugger is hard since the classes provide virtually no information that users can relate to the source code.
It is hard to debug exceptions coming from Arbitratries, and it is hard to understand the contents of Java debugger.
Suggested Solution
Add a mode where ArbitraryImpl, Combinator, etc would capture stacktrace (or just the most top frame excluding certain library-specific ones), so Arbitrary#toString could provide the source code location where the arbitrary was created.
An additional or alternative approach could be adding displayName or something like that to Arbitrary, Combiner, so users could provide meaningful names.
There's a similar case in Google Guice. When Guice identifies a configuration error, it attempts to provide the exact line in the user code which triggered the error. They do that by adding their own stacktrace filtering: https://google.github.io/guice/api-docs/latest/javadoc/com/google/inject/Binder.html#skipSources(java.lang.Class...)
Discussion
Collecting stack traces always might be time-consuming, so I suggest adding a feature flag. I'm not sure if there's a good way to require displayName from the end-user without redoing all the jqwik API.
JavaScript treats .displayName for naming "anonymous functions" for quite some time now: https://bugs.chromium.org/p/chromium/issues/detail?id=17356, https://rreverser.com/gh-stack-displayname/
A relavant issue: net.jqwik.engine.properties.arbitraries.randomized.IgnoreExceptionGenerator@4e4395c missed more than 10000 times.
My current workaround for "missed more than 10000 times" is to add maxMisses parameter with a random (hard-coded) value, so when I see Filtering [net.jqwik.api.RandomGenerator$$Lambda$497/0x000000080042a440@6391b276] missed more than 934 times I can check for 934 in my code.
Add a mode where ArbitraryImpl, Combinator, etc would capture stacktrace (or just the most top frame excluding certain library-specific ones), so Arbitrary#toString could provide the source code location where the arbitrary was created.
That means it would have to be captured on each creation of an arbitrary, right? And then the (potentially filtered) stack trace used in toString()? Quite some overhead so using a property attribute with a default would definitely be required.
That means it would have to be captured on each creation of an arbitrary, right? And then the (potentially filtered) stack trace used in toString()?
Exactly.
Quite some overhead
That is why I think it should be disabled by default, and it should be activated on demand only, or only when jqwik detects a failure.
For instance, if jqwik is about to print the final results, it might re-execute the test with debugging mode, so the output stacktrace is easier to understand.
An additional or alternative approach could be adding displayName or something like that to Arbitrary, Combiner, so users could provide meaningful names.
Something like that has been in my backlog for a while. There're two reasons that I've refrained from doing it:
- It would require an implementation for each and every arbitrary to be useful
- Some interesting stuff just cannot be captured at runtime, like the concrete type of a type variable or the actual predicate of a filtering lambda. So the description string would only be so useful.
For instance, if jqwik is about to print the final results, it might re-execute the test with debugging mode, so the output stacktrace is easier to understand.
It wouldn't even have to re-execute the full property, just the arbitrary creation phase should suffice.