oci-hdfs-connector icon indicating copy to clipboard operation
oci-hdfs-connector copied to clipboard

Fix test flakniess due to order dependency from `getDeclaredMethods()`

Open kaiyaok2 opened this issue 3 years ago • 0 comments

Description The tests in CachingObjectStorage.java dealing with multiple cached requests can fail nondeterministically when running in different JVMs. The issue is found using the command mvn edu.illinois:nondex-maven-plugin:1.1.2:nondex under the hdfs-connector directory after building dependencies.

Reason for Flakiness The tests call Mockito.verify() to assure that the mock client calls the routine getObject() only once when same requests are cached. However, when running in different environments, these verifications may occassionally fail. The reason is that in CachingObjectStorage.getObject(), the code calls Cache.put() with a key-value pair to update the cache. The method will create a new key-value pair ( rather than replacing the old value with the input value) if the input key (hashcode-wise) is not equal to any keys in the cache. However, in the hashcode() method, the final object to be hashed is an Array, which preserves order. While the output of getDeclaredMethods() does not guarantee any order, the array for hashing (line 779 at CachingObjectStorage.java) may have nondeterministic element orders. As a result, the hashcode of two same keys can be different, leading to "duplicated" entries in the cache, and the mock client can then call getObject() more times than desired.

List of flaky tests in CachingObjectStorage.java

  • testPrepopulateCache()
  • testTwoGetsCached()
  • testTwoGets_Cached()
  • testTwoGets_Cached_StrongConsistency()
  • testTwoGets_Cached_StrongConsistency_ifMatch()
  • testTwoGets_Cached_StrongConsistency_ifMatch_changed()
  • testTwoDistinctGets_StrongConsistency()
  • testTwoDistinctGets_StrongConsistency_ETagChanges()

Sample Failure

Failed tests:
  CachingObjectStorageTest.testTwoGetsCached:243
mockClient.getObject(<any>);
Wanted 1 time:
-> at com.oracle.bmc.hdfs.caching.CachingObjectStorageTest.testTwoGetsCached(CachingObjectStorageTest.java:243)
But was 2 times.

Proposed Fix Convert the array into a normal HashSet before hashing to avoid unwanted order dependency.

OCA signed https://oca.opensource.oracle.com/api/v1/oca-requests/6710/documents/6730/download (approved)

kaiyaok2 avatar Apr 14 '22 15:04 kaiyaok2