BackstopJS Not Detecting Missing UI Elements Despite Zero Mismatch Threshold - False Positive Issue

[biswabose1992]

BackstopJS Not Detecting Missing UI Elements Despite Zero Mismatch Threshold - False Positive Issue

Issue Description

BackstopJS is producing false positives by failing to detect visual regressions when UI elements are completely missing from the test image compared to the reference image, even with misMatchThreshold set to 0.

Expected Behavior

When misMatchThreshold is set to 0, BackstopJS should detect any visual differences between the reference and test images, including missing UI elements. A completely missing "Send feedback" button should result in a test failure.

Actual Behavior

BackstopJS reports the test as PASSING (false positive) when a significant UI element is completely missing from the test image. The "Send feedback" button is entirely absent in the test image but present in the reference image, yet the comparison passes.

Configuration Tested

{
  "misMatchThreshold": 0,
  "requireSameDimensions": false
}

Also tested with:

{
  "misMatchThreshold": 0,
  "requireSameDimensions": true
}

Steps to Reproduce

1. Set up BackstopJS configuration with misMatchThreshold: 0 and requireSameDimensions: false
2. Create a reference image with complete UI including the "Send feedback" button
3. Generate a test scenario where the "Send feedback" button is missing/removed
4. Run BackstopJS visual comparison
5. Result: Test passes despite obvious missing button

Environment

• BackstopJS Version: [6.3.25] Also tried out with 6.2.2
• Browser: [Chromium]
• OS: [MAC OS]

Detailed Findings

With requireSameDimensions: false (Current Issue)

• Test passes despite missing "Send feedback" button
• No visual regression detected
• Images have different heights due to missing button
• This is the main issue - false positive

With requireSameDimensions: true (Also Problematic)

• Test fails due to dimension mismatch
• BUT fails for wrong reason (dimension difference, not missing button)
• Doesn't specifically identify what visual content is different
• Catches the height change but not the semantic difference

Screenshots

Core Problem

BackstopJS appears to have issues with:

1. Pixel-by-pixel comparison when images have different dimensions
2. Missing content detection at page boundaries

Questions for Maintainers

1. How does BackstopJS handle pixel comparison when images have different heights?
2. Is this expected behavior when requireSameDimensions: false?

Expected Fix/Enhancement

BackstopJS should detect missing UI elements as visual regressions regardless of dimension differences when misMatchThreshold is set to 0. The comparison algorithm should account for missing buttons and other UI elements, flagging them appropriately.

[garris]

Thank you for this clear bug report. This definitely sounds wrong. Threshold of 0 should do an exact comparison and thus find any difference.

requireSameDimensions:false skips a check at the beginning that simply checks image size. Non overlapping areas are considered "different"

Hope that helps.

[biswabose1992]

What do you mean by "Non overlapping areas are considered "different"