Visual diffs and reporters in Hardy

[dwabyick]

We're (@frio80 and I) trying to use Hardy for visual regression testing on a JS/CSS UI library.

Hardy gets us part of the way there by providing a tool that will 'fail' on a visual difference, but we haven't figured out how to create 'reports' on what changed.

The use-case is for developers iteratively building locally, and us giving them more information on what their changes effects. For example, if they visually change Component X, lets let them know if it changes Components Y and Z. They may have expected that change, they may not have.

It seems that we'd a couple of features to make this happen:

• Add the ability to create the visual diff in Hardy to get this to happen. (I've experimented with this in my dwabyick fork).
• Create a report of all failed tests from Hardy. This seems like a general Cucumber runner question, maybe. I'm new to Cucumber.
• A third 'maybe' feature would be more explicit support in Hardy for creating a baseline. Its kind of an implicit thing that happens the first time you run Hardy. It could help make it more intuitive for this use case.

Are these use-cases interesting to you? Happy to help as I can.

[thingsinjars]

This definitely sounds like a good avenue to explore. I started wondering about reporters just this weekend (actually after discovering the mocha nyan reporter). Along with the plugin API ideas, there's a lot to think about.

I'll start off with your ImageMagick fork and see if there's a simple way to specify the reporter in Cucumber.

[thingsinjars]

The options for reporter from Cucumber are

• JSON (looks like this: https://gist.github.com/thingsinjars/7088966 )
• Pretty (Writes out the scenario)
• Progress (the current one)
• Summary (the last lines of Progress)

It'd be straightforward to modify Hardy-CLI to read that from a flag or from the config file.

[dwabyick]

cool! that will help.

What about continuing tests after failures? I read a bit about this, and it seems like you just have to write a contrived Cucumber test, perhaps, to do this. Is that right?

I also wonder if its possible to get failed data selector data into the JSON output. Seems possible via the callback. I'll look into that.