ConcurrentModificationException in cookie handling

[celandro]

Unfortunately this is the longest threaddump I have: at org.sonatype.spice.jersey.client.ahc.AhcClientHandler.handle(AhcClientHandler.java:137) ~[jersey-ahc-client-1.0.5.jar:na] at com.sun.jersey.api.client.Client.handle(Client.java:648) ~[jersey-client-1.12.jar:1.12] at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680) ~[jersey-client-1.12.jar:1.12] at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.12.jar:1.12] at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:558) ~[jersey-client-1.12.jar:1.12] .... removed .... Caused by: java.util.ConcurrentModificationException: null at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859) ~[na:1.7.0_45] at java.util.ArrayList$Itr.remove(ArrayList.java:845) ~[na:1.7.0_45] at org.sonatype.spice.jersey.client.ahc.AhcClientHandler.applyResponseCookies(AhcClientHandler.java:154) ~[jersey-ahc-client-1.0.5.jar:na] at org.sonatype.spice.jersey.client.ahc.AhcClientHandler.handle(AhcClientHandler.java:125) ~[jersey-ahc-client-1.0.5.jar:na]

[celandro]

This ends up killing the connection pool by the way. The next exceptions are: com.sun.jersey.api.client.ClientHandlerException: java.lang.NullPointerException at org.sonatype.spice.jersey.client.ahc.AhcClientHandler.handle(AhcClientHandler.java:137) ~[jersey-ahc-client-1.0.5.jar:na] at com.sun.jersey.api.client.Client.handle(Client.java:648) ~[jersey-client-1.12.jar:1.12] at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680) ~[jersey-client-1.12.jar:1.12] at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.12.jar:1.12] at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:558) ~[jersey-client-1.12.jar:1.12] ..... removed.... Caused by: java.lang.NullPointerException: null at com.ning.http.client.cookie.CookieEncoder.encode(CookieEncoder.java:26) ~[async-http-client-1.8.10.jar:na] at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.construct(NettyAsyncHttpProvider.java:821) ~[async-http-client-1.8.10.jar:na] at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.buildRequest(NettyAsyncHttpProvider.java:650) ~[async-http-client-1.8.10.jar:na] at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.buildNettyResponseFutureWithCachedChannel(NettyAsyncHttpProvider.java:962) ~[async-http-client-1.8.10.jar:na] at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.doConnect(NettyAsyncHttpProvider.java:1014) ~[async-http-client-1.8.10.jar:na] at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.execute(NettyAsyncHttpProvider.java:935) ~[async-http-client-1.8.10.jar:na] at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:512) ~[async-http-client-1.8.10.jar:na] at org.sonatype.spice.jersey.client.ahc.AhcClientHandler.handle(AhcClientHandler.java:123) ~[jersey-ahc-client-1.0.5.jar:na]

Followed by com.sun.jersey.api.client.ClientHandlerException: java.io.IOException: Too many connections 10 at org.sonatype.spice.jersey.client.ahc.AhcClientHandler.handle(AhcClientHandler.java:137) ~[jersey-ahc-client-1.0.5.jar:na] at com.sun.jersey.api.client.Client.handle(Client.java:648) ~[jersey-client-1.12.jar:1.12] at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680) ~[jersey-client-1.12.jar:1.12] at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.12.jar:1.12] at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:558) ~[jersey-client-1.12.jar:1.12] ---removed--- Caused by: java.io.IOException: Too many connections 10 at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.doConnect(NettyAsyncHttpProvider.java:1057) ~[async-http-client-1.8.10.jar:na] at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.execute(NettyAsyncHttpProvider.java:935) ~[async-http-client-1.8.10.jar:na] at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:512) ~[async-http-client-1.8.10.jar:na] at org.sonatype.spice.jersey.client.ahc.AhcClientHandler.handle(AhcClientHandler.java:123) ~[jersey-ahc-client-1.0.5.jar:na]

[slandelle]

Caused by: java.lang.NullPointerException: null at com.ning.http.client.cookie.CookieEncoder.encode(CookieEncoder.java:26)

If I check out CookieEncoder.java:26, you obviously have a null Cookie in the cookie list.

How you ended up with this is probably for you to figure out.

A possibility could be that you're keeping a reference to a RequestBuilder, send the Request, and then try to edit the RequestBuilder. If so, check out RequestBuilder code and you'll realize that RequestBuilder is neither threadsafe nor immutable and share its state with the Request it builds. It's just a fluent API.

[celandro]

We do not reuse requestbuilder in our production code.

 @Override
 public <R, T> R post(final String url, final T data, final Class<R> c) {
     final WebResource webResource               = restfulClient.resource(url);
     final WebResource.Builder requestBuilder    = webResource.getRequestBuilder();

     final Class<? extends Object> dataClass = data.getClass();
     final ClientResponse response = applyType(applyAccept(requestBuilder), dataClass).entity(data).post(ClientResponse.class);
     try {
         return getEntity(c, response);
     } finally {
         closeResponse(response);
     }
 }

[celandro]

PS. Please check the pull request. The new unit test throws the ConcurrentModificiationException.

I do not know what causes this exception to eventually kill the connection pool but on all 9 servers, it occurs first and eventually they all run out of connections and never recover. Some get the NullPointerException and some do not. Some have 1 ConcurrentModificiationException, some have 6

[celandro]

Ps. All cookie related code is only inside ahc jersey client. We do not set any cookies ourselves. They are only returned by the server, saved by AhcClientHandler and set in the request by AhcClientHandler.

The bug fix is simply to replace non-threadsafe linkedlist with a threadsafe collection.