Inconsistent test results using v5.3.0 or v.5.0.7 with no docker (similar to #1303)

[akmil]

When I compare some url with itself I have some random fail/pass results.

Here my config file:

module.exports = {
              id: "uniq_id_config",
              viewports: [
                  {
                      label: "tablet",
                      width: 1024,
                      height: 768
                  }
              ],
        onBeforeScript: "puppet/onBefore.js",
        onReadyScript: "puppet/onReady.js",
        scenarios: [
            {
                label: 'Home',
                misMatchThreshold : 0.5,
                referenceUrl: 'https://dewalt.com/',
                url: 'https://dewalt.com/' 
                }
            ],
        paths: {
            bitmaps_reference: "backstop_data/bitmaps_reference",
            bitmaps_test: "backstop_data/bitmaps_test",
            engine_scripts: "backstop_data/engine_scripts",
            html_report: "backstop_data/html_report",
            ci_report: "backstop_data/ci_report"
        },
        report: ["browser"],
        engine: "puppeteer",
        engineOptions: {
            args: ["--no-sandbox"]
        },
        asyncCaptureLimit: 5,
        asyncCompareLimit: 50,
        debug: false,
        debugWindow: false
}

I'm using 'https://dewalt.com/' to test. I've tested with other url like: 'https://itc.ua/' - it have the same behavior. Also tested with the last version of backstopjs - it have the same behavior.

link to repo: https://github.com/akmil/backstop-demo

[r-oldenburg]

Could this be related to https://github.com/garris/BackstopJS/issues/1344?

[klodoma]

Please give it first a try with asyncCaptureLimit: 1 and let us know how that goes.

I am also experiencing inconsistent results over web-pages and still not sure where that comes from.