ConcurrentModificationException :: Iterating through map while expirator is running

[expiringmapuser]

Hello,

For my use case of expiring map I see a ConcurrentModification exception when using an Expirator with policy CREATED. The issue occurs when an iteration is made on the map and an expirator is running:

    java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:711)
at java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:744)
at java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:742)
at net.jodah.expiringmap.ExpiringMap$EntryLinkedHashMap$AbstractHashIterator.getNext(ExpiringMap.java:364)
at net.jodah.expiringmap.ExpiringMap$EntryLinkedHashMap$EntryIterator.getNext(ExpiringMap.java:385)
at net.jodah.expiringmap.ExpiringMap$EntryLinkedHashMap$EntryIterator.next(ExpiringMap.java:387)
at net.jodah.expiringmap.ExpiringMap$EntryLinkedHashMap$EntryIterator.next(ExpiringMap.java:385)
at net.jodah.expiringmap.functional.ConcurrencyTestWithExpirator.deleteWithEntrySetIterator(ConcurrencyTestWithExpirator.java:40)`

This happens with both EntrySet iterator, or values/keyset iterators.

Note: I read about issue https://github.com/jhalterman/expiringmap/issues/10 that is why I tried with EntrySet too.

Thank you!

[jhalterman]

@expiringmapuser Do you have some code you can share that reproduces this?

[expiringmapuser]

This is a test that can show the problem:

• It is not not concurrency proof to replicate irrespective of the system/processing power. This is why it may require tweaking for different values like the max values in map/sleep times/expiration interval
• It allows testing with both EntrySet or values iterators
• Logging uses the console. Adding a logging library required to also upload the main pom and I preferred to only upload the test case.

package net.jodah.expiringmap.functional;

import net.jodah.expiringmap.ExpirationListener;
import net.jodah.expiringmap.ExpirationPolicy;
import net.jodah.expiringmap.ExpiringMap;
import org.testng.annotations.Test;

import java.util.Iterator;
import java.util.Map.Entry;
import java.util.concurrent.TimeUnit;

public class ConcurrencyTestWithExpirator {
    private static final boolean LOG_ENABLED = false;
    private static final int MAX_ENTRIES_IN_MAP = 2400000;
    private static final long SLEEP_TIME_MILLIS = 1L;
    private static final int EXPIRATION_SECONDS = 1;

    @Test
    public void shouldSupportConcurrentExpiratorWithIteration() throws InterruptedException {
        ExpiringMap<Integer, Integer> expiringMap = expiringMap();
        addEntriesTo(expiringMap);

        deleteWithEntrySetIterator(expiringMap);
//        deleteWithValuesIterator(expiringMap);

    }

    private void deleteWithValuesIterator(ExpiringMap<Integer, Integer> expiringMap) throws InterruptedException {
        Iterator<Integer> iterator = expiringMap.values().iterator();
        while (iterator.hasNext()) {
            Integer mapValue = iterator.next();
            log("Deleting " + mapValue);
            iterator.remove();
            arbitrarilySleep();
        }
    }

    private void deleteWithEntrySetIterator(ExpiringMap<Integer, Integer> expiringMap) throws InterruptedException {
        Iterator<Entry<Integer, Integer>> iterator = expiringMap.entrySet().iterator();
        while (iterator.hasNext()) {
            Entry<Integer, Integer> mapEntry = iterator.next();
            log("Deleting " + mapEntry.getKey());
            iterator.remove();
            arbitrarilySleep();
        }
    }

    private ExpiringMap<Integer, Integer> expiringMap() {
        return ExpiringMap.builder()
                .expirationPolicy(ExpirationPolicy.CREATED)
                .expiration(EXPIRATION_SECONDS, TimeUnit.SECONDS)
                .expirationListener(loggingExpirationListener())
                .build();
    }

    private ExpirationListener<Integer, Integer> loggingExpirationListener() {
        return (key, value) -> {
            log("Expiring " + key + " ");
        };
    }

    private void arbitrarilySleep() throws InterruptedException {
        Thread.sleep(SLEEP_TIME_MILLIS);
    }

    private void log(String logMsg) {
        if (LOG_ENABLED) {
            System.out.println(logMsg);
        }
    }

    private void addEntriesTo(ExpiringMap<Integer, Integer> expiringMap) {
        for (int i = 0; i < MAX_ENTRIES_IN_MAP; i++) {
            expiringMap.put(i, i);
        }
    }
}

[harrydevane]

:+1:

[charlesvien]

👍

[tobyxdd]

Same problem here, any updates? 😢

[Carpetfizz]

Having the same exact issue...

[arivazhagan-jeganathan]

Hi, do we have any solution available to overcome this Exception? when can we expect a fix for this? Please let us know

[narendraj9]

@jhalterman I am trying to understand the reason for this issue. https://github.com/jhalterman/expiringmap/blob/77d217ba7939f0712c3496c56478341fac5e01d3/src/main/java/net/jodah/expiringmap/ExpiringMap.java#L359

Shouldn't we guard the above with readLock.lock()?

[mishaxz]

I have this problem too and #10 fix doesn't help. Mine is ExpirationPolicy.ACCESSED though

note to self: #153358916

[narendraj9]

@mishaxz We had to move to Guava cache because of this.

[hstr0100]

Moved from Guava (and Caffeine) due to the inconsistency of getIfPresent() under heavily multi-threaded environments, and now instead I got this exact same problem reported back in 2016.

Initially I didn't see the need to make my own implementation of a concurrent self-expiring map considering there were some light-weight out of the box alternatives, but now it's basically the only option I have

[ben-manes]

What do you mean by inconsistency of getIfPresent() under heavily multi-threaded? One would expect entries to expire with some threads seeing it and some not. In both libraries the use of getIfPresent() is disfavored compared to a computing get.

[jhalterman]

Fixed by #78

[jhalterman]

This has been fixed in 0.5.11.