BackstopJS

BackstopJS copied to clipboard

Hello!

First off, thanks again for an amazing project here - having now used BackstopJS for a few weeks, it’s paid off in a big way and become an essential part of our workflow at Snapppt.

we develop on a Mac and use CircleCI for our continuous integration setup - on which we’re using a Ubuntu Trusty image. The reference screenshots are being recorded on a mac. Having successfully got Backstop running as part of our build on CI, the issue is that Backstop tests are failing due to a difference in font rendering. The text on Mac is rendering a little more bold, and a little shorter than it is on Ubuntu - which is causing the text itself to be picked up as different by Backstop, but more significantly, the slight height different is pushing other content down by a tiny amount and resulting in a larger mismatch.

Mac on left, Ubuntu in middle - with font rendering difference causing a diff in large parts of the image

So my question here is - assuming many will have a similar workflow to ours, what methods are you using to address the difference in font rendering between the two environments?

It seems there’s a few options. I thought I’d reach out to hear if anyone has already tackled this in a reliable manner. Some options I see include:

1. Modifying Ubuntu’s font rendering to match Mac OSX - Through either editing fontConfig files, or making use of the infinality project. Having spent a little time looking into this, I’ve had little luck thus far - and wonder if it’d even be possible to get close enough to Mac in order to pass a Backstop test.

2. Creating the Backstop reference screenshots in Ubuntu locally - This would likely be via a Docker container or perhaps a VirtualBox saved Ubtunty Trusty setup. This would make running with Backstop less convenient, and also means the Ubuntu view would be our reference - when we’d prefer Mac to be the definitive recorded view, given it’s both what the platform is developed on, and how it is viewed by the majority of our users.

3. Having our CI run on an OSX server - It appears CircleCI supports running builds on OSX machines which could very well do away with this issue altogether. The downside to us there would be that we lose the benefit of our other tests (E2E, unit tests, backend tests etc) running on a server closer to that of production (which are also on linux servers).

A pretty general question here - but I’d welcome any thoughts on how others are addressing the issue of the font rendering differences environments causing failures in Backstop tests.

I think you have a few options on top of what you've already mentioned. Unfortunately, all of them are compromises of some kind or other, and you'll have to find the combination that works best for you and your project.

• You could raise misMatchThreshold to make your tests ignore small differences in font rendering.
• You could add a little bit of extra styling to make your layout move around less, or change your testing selectors to test parts that change the most separately from parts that change the least.
• You could change your font to one that produces less rendering differences (maybe just for testing), or hide elements that produce rendering differences.
• You could generate two sets of reference screenshots, one for each env.

@mrjamesriley Thanks for posting this in detail! I know this is a non-trivial issue for a few developers who have also posted similar questions here from time to time.

So, you've probably tried the following already but -- just a few suggestions incase you haven't...

Have you tried disabling anti-aliasing during testing? e.g.

// file backstop_data/casper_scripts/myBeforeScript.js
module.exports = function(casper, scenario, vp) {
  document.getElementsByTagName('body')[0].style['-webkit-font-smoothing'] = 'none';
}

Does that normalize the font such that your differentials are eliminated? (I haven't tried it myself so I really don't know) If it works though, I would think you could have confidence that your app behaves the same across environments. Which I think is what I think you're after.

Another approach would be to take multiple screen shots so vertical noize doesn't pile up. IOW instead of snapping the whole document you could capture say .row1, .row2, '.row3'. You would still have to fudge the mismatch number -- but only by a little bit since error noise wouldn't accumulate as page depth increases.

Please post back here when you decide which way to go! Thanks!

Any news about this? I still have this problem.

I am currently doing battle with this issue. References generated on OSX fail on Jenkins/CI Ubuntu, in my case raising the threshold to 6.45 and setting requireSameDimensions to false gets the test passing, but this doesn't seem ideal.

Removing -webkit-font-smoothing (as suggested above) changes the reference grabs but doesn't affect the outcome, the fonts are still rendered differently (just different differently).

I want to avoid having to run a local VM to generate the references, but struggling to find an alternative & acceptable configuration based workaround.

@garris any further ideas welcome!

Hello @richhiggins ,

Just out of curiosity. Why not working on a live / same dev environment for keeping reference images and test images both ?

@mirzazeyrek our team work on OSX locally and that's not something up for grabs. Likewise, the server is Ubuntu and cannot be OSX.

Whilst installing, configuring and maintaining a second local dev environment is certainly technically possible it's also undesirable. Trying hard to keep things simple :)

@richhiggins Is your ubuntu server live ? Why not using it for creating reference images ? Are you keeping reference images on git repo ?

Hi @richhiggins. This is actually a big problem for us now as well. We have multiple deployment environments and are wanting a way to test our deploys from one stage to the next. The problem is obviously, different environments have different rendering characteristics.

For instance, a retina laptop will actually render differently when attached to standard resolution monitors. And then moving to different platforms/configurations complictes things more.

I really don't want to have to resort to a VM either.

So, still working on a real fix. There is now a chrome-headless branch of backstop -- which is awesome (but still not ready for GM) -- I am hoping to find a way to address the issue there.

Will keep you posted.

@mirzazeyrek thanks for keeping an eye on this! In my case it's really just a convenience issue. We wanted any developer to be able to run locally. And also it would be great to avoid having to put in the effort to create/distribute/maintain a VM or backstop server. That said -- we will totally go that route if we can't get a reliable solution.

At the moment, generate our reference screenshots on the CI server and then download them and put them in source control. We then know when CI does a test, the same environment that generated the references is being used. Not ideal but holding out for Chromy support 😄

@garris If you have a moment I would like to discuss about this issue by showing actual use case. Please add me on skype mirza.zeyrek

@jtuds yes -- your solution is very close to what we are more or less doing in some cases.

I'm also hoping we can find a magic switch in chrome! 😉

Does chrome has same font rendering for mac os and ubuntu and windows ?

I have no idea at the moment. I will attempt to reach out to someone on the chrome tools team to see if I can get some guidance.

I have also seen related issues posted to Gemini and Casper boards. I know we are not he only ones with this issue.

Hey! We had the same problem. How we solve it:

• using snapshots as HTML pages and re-render them each time Another option:
• using docker to run visual tests

I've not got this 100% up and running yet, but the approach I've settled on is keeping 2 lots of references, production (linux) & development (OSX), branching the configuration and using a Jenkins task to update the production references as required.

Ideally the production references update will be triggered by committing updated development references, via a dedicated branch or a commit message if possible... TBC

Thought it might be worth sharing the approach at least.

Not a solution for everybody, but this workaround works for us: We use Github Actions and that supports MacOS. All developers run MacOS, so we decided to change the workflow file to use MacOS instead of Ubuntu: runs-on: macos-latest.