assertNotEquals is incorrect for comparing Collection instances

[stuart-marks]

TestNG Version

The testng-7.5-20210913.194404-31 snapshot, but this also occurs in previous releases including 7.4.0, 7.3.0, and 6.9.5.

Expected behavior

assertNotEquals(Collection, Collection) should compare the collections element by element, similar to assertEquals(Collection, Collection)

Actual behavior

assertNotEquals(Collection, Collection) delegates to the first argument's equals() method, which can return a result that's not the inverse of assertEquals

Is the issue reproducible on runner?

• [x] Shell
• [ ] Maven
• [ ] Gradle
• [ ] Ant
• [ ] Eclipse
• [ ] IntelliJ
• [ ] NetBeans

Test case sample

I've continued to look at the equality asserts, especially those related to collections, as a followup to #2540. The code paths for the collection overloads of assertEquals and assertNotEquals are quite a bit different, so perhaps it isn't a surprise that they end up not comparing the same thing. This wasn't introduced at the same time as the 7.4.0 breakage though; apparently this has been in the code base for a long time. I haven't had the time to investigate the history any further than running it on several binaries I had available.

Here's a simple test case:

import org.testng.annotations.*;
import static org.testng.Assert.*;

import java.util.*;

@Test
public class AssertNotEqualsCollection {
    Collection<String> c1;
    Collection<String> c2;

    @BeforeClass
    public void setup() {
        c1 = List.of("a", "b", "c");
        c2 = new LinkedHashSet<>(c1);
    }

    // This test passes, as expected.
    @Test
    public void checkEquals() {
        assertEquals(c1, c2);
    }

    // This test also passes!!
    @Test
    public void checkNotEquals() {
        assertNotEquals(c1, c2);
    }
}

The checkEquals test passes, as expected. It calls assertEquals which compares the collections' contents element by element.

The checkNotEquals test also passes, unexpectedly. Inspecting assertNotEquals reveals that it follows a different path, going through getNotEqualReason. This does some reference comparison and null checks and then ends up calling Objects.equals(actual, expected). This JDK method ends up calling actual.equals(expected). This returns false because the logic in the collections checks the runtime types, and a List can never be equal to a Set -- even if they contain the same elements in the same order. Thus assertNotEquals returns without throwing, and checkNotEquals also passes.

There's no documentation for assertNotEquals but one would expect that it always asserts the inverse of the corresponding assertEquals method.

[juherr]

Thanks for the report. Strange we didn't see it before. We will check it asap.