Use Guava's testlib for increased test coverage

[ben-manes]

Guava provides a convenient testlib for their collection tests. This can be used for other collection implementations, making a handy second set of tests to catch oversights. For example it found a few small bugs in FastUtil. In the test case below the Map.entrySet().toString() is missing, so it does not pretty-print the contents (this can be ignored using suppressing(Method)).

Maps.mutable tests

import java.util.Map;
import java.util.function.Supplier;

import org.eclipse.collections.api.factory.Maps;

import com.google.common.collect.testing.NavigableMapTestSuiteBuilder;
import com.google.common.collect.testing.TestStringMapGenerator;
import com.google.common.collect.testing.features.CollectionFeature;
import com.google.common.collect.testing.features.CollectionSize;
import com.google.common.collect.testing.features.MapFeature;

import junit.framework.Test;
import junit.framework.TestCase;

/**
 * @author [email protected] (Ben Manes)
 */
public class EclipseMapTest extends TestCase {

  public static Test suite() {
    return suite("Maps.mutable", Maps.mutable::empty);
  }

  public static Test suite(String name, Supplier<Map<String, String>> factory) {
    return MapTestSuiteBuilder
        .using(new TestStringMapGenerator() {
          @Override protected Map<String, String> create(Map.Entry<String, String>[] entries) {
            var map = factory.get();
            for (var entry : entries) {
              map.put(entry.getKey(), entry.getValue());
            }
            return map;
          }
        })
        .named(name)
        .withFeatures(
            CollectionSize.ANY,
            MapFeature.GENERAL_PURPOSE,
            MapFeature.ALLOWS_NULL_KEYS,
            MapFeature.ALLOWS_NULL_VALUES,
            MapFeature.ALLOWS_NULL_ENTRY_QUERIES,
            CollectionFeature.SUPPORTS_ITERATOR_REMOVE)
        .createTestSuite();
  }
}

junit.framework.ComparisonFailure: emptyCollection.toString should return [] expected:<[[]]> but was:<[org.eclipse.collections.impl.map.mutable.UnifiedMap$EntrySet@0]>
	at junit.framework.Assert.assertEquals(Assert.java:100)
	at junit.framework.TestCase.assertEquals(TestCase.java:253)
	at com.google.common.collect.testing.testers.CollectionToStringTester.testToString_size0(CollectionToStringTester.java:49)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at junit.framework.TestCase.runTest(TestCase.java:177)
	at junit.framework.TestCase.runBare(TestCase.java:142)
	at junit.framework.TestResult$1.protect(TestResult.java:122)
	at junit.framework.TestResult.runProtected(TestResult.java:142)
	at junit.framework.TestResult.run(TestResult.java:125)
	at junit.framework.TestCase.run(TestCase.java:130)
	at junit.framework.TestSuite.runTest(TestSuite.java:241)
	at junit.framework.TestSuite.run(TestSuite.java:236)
	at junit.framework.TestSuite.runTest(TestSuite.java:241)
	at junit.framework.TestSuite.run(TestSuite.java:236)
	at junit.framework.TestSuite.runTest(TestSuite.java:241)
	at junit.framework.TestSuite.run(TestSuite.java:236)
	at junit.framework.TestSuite.runTest(TestSuite.java:241)
	at junit.framework.TestSuite.run(TestSuite.java:236)
	at junit.framework.TestSuite.runTest(TestSuite.java:241)
	at junit.framework.TestSuite.run(TestSuite.java:236)
	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:90)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:93)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:40)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:529)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:756)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:452)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:210)

[kevinb9n]

Whoa, just happened to see this 15 hours later.

Note that these suites are extremely picky. But as you can see, you can turn things off by feature, and you can also suppress individual tests.

Some of the picky stuff doesn't matter much in absolute terms, but it makes migrating to your collections from others a more "guaranteed safe" operation.

It would be nice if we could rewrite it using modern JUnit instead of hacking the suites together the way we do. It would just be a project.

[donraab]

Thank you for sharing this @ben-manes! I was not aware of this project until I saw this issue. Definitely agree with @kevinb9n comment on helping make migrations "guaranteed safe" operations. Thank you both for the comments and pointers!

[ben-manes]

@kevinb9n that reddit thread reminded me to go take a peek at EC, and I am now porting EC's mutable map unit tests into Caffeine as another sanity check. On quite a few projects I have run the Guava's tests out of curiosity and found small bugs (spring, expiringmap, cache2k, primitive collections, fastutil, coherence), so it is very nice but unknown gem.

Caffeine has its own map tests, but like others is not designed for reuse. Those use parameterized testing to run against every cache configuration the matches the specification constraints. I know in TestNG you can use an @Factory for dynamic tests like the TestSuiteBuilders, but I am not familiar enough with JUnit 5 except that it seems to be far closer to TestNG than to JUnit 3/4. Of course I also use Guava's map tests, too.

[ben-manes]

@donraab I am not sure if you have custom concurrent collections, but Lincheck is nice for linearization testing (example).

[donraab]

@ben-manes We have a custom ConcurrentHashMap (and an unsafe version of same). We also have MultiReader collections for List, Set, Bag. Thank you for the link, I will check it out!

[Speiger]

o/ Just found this one (Author of Primitive Collections) I have created a template of "Guavas Unit Testing Library" that basically generates all permutations for Collections (Primitive Types) While a lot of work, it managed to get me to 80% coverage of all permutations (it is an average) and I am trying to make this like a consistent average across all permutations (sets/lists/queues are still lacking a bit).

But if you want to get some inspirations on how to get to that level too feel free to look. https://github.com/Speiger/Primitive-Collections/tree/debug/src/builder/resources/speiger/assets/testers/templates

(Note that this is like a custom template library that I have written myself that basically can do very basic "if" statements & "ignore" areas where certain lines can be ignored from the templater)

Small note: don't try to run the unit tests, these take an hour on average to complete....