Jersey Apache Connector Hangs …?

Jersey comes with various connectors to third-party HTTP clients. The way the connector is used is simple, put the connector and the third-party client to the classpath, and tell the client to use it. For Apache Connector, use:

ClientConfig clientConfig = new ClientConfig();
clientConfig.connectorProvider(new ApacheConnectorProvider());
Client client = ClientBuilder.newClient(clientConfig);

Switching from the default HttpUrlConnectorProvider to ApacheConnectorProvider may not be as smooth as expected, and the connection can hang. Surprisingly, even switching the provider in Jersey tests and examples in Jersey workspace. For instance, adding Apache connector to org.glassfish.jersey.tests.e2e.sse.BroadcasterCloseTest can result in a thread lock similar to:

java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000ee799320> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:379)
 at org.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69)
 at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:245)
 - locked <0x00000000ed700070> (a org.apache.http.pool.AbstractConnPool$2)
at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:304
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
 at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
 at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
 at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
 at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
 at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
 at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:478)

The problem is the test code, the response is never closed. The Apache Connection pool does not have enough threads to execute all the requests since the response is not closed and thus the connection is not closed. The solution can be simple, call response.close() after the assertion in the for-each loop as well as for 2 more responses. Note that the third request is closed automatically when response.get(String.class) is called.

The general rule is to close anything that can be closed if not needed (Client, Response, SseEventSink, SseEventSource). Response.close() does not close the buffered entity, it can be read multiple times. If the entity is an InputStream, do not forget to close it, too.

Sometimes, the response cannot be closed, yet, and multiple connections need to be open. This time, the Apache Connector needs to be provided with more threads. For BroadcasterCloseTest, eleven is enough (in case the responses are not closed):

@Override
protected void configureClient(ClientConfig config) {
    PoolingHttpClientConnectionManager cm
            = new PoolingHttpClientConnectionManager();
    cm.setMaxTotal(11);
    cm.setDefaultMaxPerRoute(11);
    config.property(
            ApacheClientProperties.CONNECTION_MANAGER , cm);
    config.connectorProvider(new ApacheConnectorProvider());
}

There is a case where the Jersey Apache Connector hangs even if everything is done properly. Apache HttpClient 4.5.1 comes with a change that may cause Jersey Apache Connector in the case where chunked streams are used. An example when this happens is StreamingTest#clientCloseTest. Jersey 2.28 depended on Apache HttpClient 4.5, and the Apache Connector did not hang with it. Jersey 2.29 came with newer Apache Connector and for the StreamingTest#clientCloseTest to pass, a timeout 1000ms has been added:

 client.property(ClientProperties.READ_TIMEOUT, 1_000);

Without the timeout, the connector hangs, and the stack trace is as follows:

java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:261)
at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:222)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:183)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:210)
at org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:312)
at org.apache.http.impl.execchain.ResponseEntityProxy.streamClosed(ResponseEntityProxy.java:142)
at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)
at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)
at java.io.BufferedInputStream.close(BufferedInputStream.java:483)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at org.glassfish.jersey.apache.connector.ApacheConnector...

Jersey 2.30 contains a workaround for the changed Apache Http Client 4.5.1+ and the timeout is not needed to be added in the client code anymore.