FactCheck.jl

FactCheck.jl copied to clipboard

Currently the only allowable operator in a @fact is =>. I have quite a few tests that do other boolean tests (e.g. check that @allocated is less than N bytes). Right now I defined a function "lessthan" that's a FactCheck-compatible test, but it's pretty verbose.

What I'd like is for the @fact macro to also handle other top-level boolean operators like <=, <, etc? Then I could just do:

@fact (@allocated my_func()) < 200

Are there any fundamental architecture issues that make this a bad idea?

This is basically what @test in Base.Test does, right? I guess the design idea here is to use functions to the right of =>. There are some of those, but there could certainly be more. In fact, I was just about to make a feature request for that, when I saw your post. My suggestion would be to implement more of these Boolean functions, perhaps along the lines of Hamcrest. Then you could, in your case, have

@fact @allocated my_func() => less_than(200)

or the like (perhaps with a different name for less_than, avoiding the underscore per Julia style guidelines; dunno). For this specific example, this might not be so useful, but for more general checks, it could be. See, for example, the Python version of Hamcrest for an example list of such matchers.

If there's any interest in this, I'd be happy to write the code. Then again, it might of course well be that @zachallaun woud want to write it himself—or to avoid adding this, for that matter. Perhaps a separate Hamcrest.jl module might even be useful, usable by other test frameworks as well. For example, Fixtures.jl by @burrowsa has a couple of matchers. There might be others as well, and I'm guessing that the fundamental nature of such matchers would be similar between frameworks.

For now, I guess adding some simple matchers what are basically currying (using lambdas) won't clutter up your test code that much. For example, I currently have the following early on in a test file:

contains(x) = y -> x in y

I then use contains(x) alongside isempty (an existing function, of course) in my tests. It would be pretty trivial to whip up similar one-liners for many Hamcrest matchers, I guess.

the @test macro just evaluates the given expression whole. The cool thing that FactCheck does with => is that it splits the left and right sides so that after a failure it can print out why the test failed.

@ssfrr As I've been refactoring/playing with the code, I've kinda got into the helper idea. At first iI was really not into it. I'm going to add in more helpers like less_than at first, but maybe a future version could shake it up operator-wise.

I feel like if you do add new operators though, => might need to go, because

@fact a => 5
@fact b != 3
@fact c >= 5
@fact d <= 5

looks a bit odd because one of those things is not like the other...

The nifty function right-hand-side thing would be odd if we allowed == because then @fact a == iseven (as perhaps odd example) would be ambiguous. Maybe this package is destined to morph into an extension of Base.Test with test grouping anyway.