Jersey Apache Connector Hangs …?

Jersey comes with various connectors to third-party HTTP clients. The way the connector is used is simple, put the connector and the third-party client to the classpath, and tell the client to use it. For Apache Connector, use:

ClientConfig clientConfig = new ClientConfig();
clientConfig.connectorProvider(new ApacheConnectorProvider());
Client client = ClientBuilder.newClient(clientConfig);

Switching from the default HttpUrlConnectorProvider to ApacheConnectorProvider may not be as smooth as expected, and the connection can hang. Surprisingly, even switching the provider in Jersey tests and examples in Jersey workspace. For instance, adding Apache connector to org.glassfish.jersey.tests.e2e.sse.BroadcasterCloseTest can result in a thread lock similar to:

java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000ee799320> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:379)
 at org.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69)
 at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:245)
 - locked <0x00000000ed700070> (a org.apache.http.pool.AbstractConnPool$2)
at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:304
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
 at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
 at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
 at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
 at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
 at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
 at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:478)

The problem is the test code, the response is never closed. The Apache Connection pool does not have enough threads to execute all the requests since the response is not closed and thus the connection is not closed. The solution can be simple, call response.close() after the assertion in the for-each loop as well as for 2 more responses. Note that the third request is closed automatically when response.get(String.class) is called.

The general rule is to close anything that can be closed if not needed (Client, Response, SseEventSink, SseEventSource). Response.close() does not close the buffered entity, it can be read multiple times. If the entity is an InputStream, do not forget to close it, too.

Sometimes, the response cannot be closed, yet, and multiple connections need to be open. This time, the Apache Connector needs to be provided with more threads. For BroadcasterCloseTest, eleven is enough (in case the responses are not closed):

@Override
protected void configureClient(ClientConfig config) {
    PoolingHttpClientConnectionManager cm
            = new PoolingHttpClientConnectionManager();
    cm.setMaxTotal(11);
    cm.setDefaultMaxPerRoute(11);
    config.property(
            ApacheClientProperties.CONNECTION_MANAGER , cm);
    config.connectorProvider(new ApacheConnectorProvider());
}

There is a case where the Jersey Apache Connector hangs even if everything is done properly. Apache HttpClient 4.5.1 comes with a change that may cause Jersey Apache Connector in the case where chunked streams are used. An example when this happens is StreamingTest#clientCloseTest. Jersey 2.28 depended on Apache HttpClient 4.5, and the Apache Connector did not hang with it. Jersey 2.29 came with newer Apache Connector and for the StreamingTest#clientCloseTest to pass, a timeout 1000ms has been added:

 client.property(ClientProperties.READ_TIMEOUT, 1_000);

Without the timeout, the connector hangs, and the stack trace is as follows:

java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:261)
at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:222)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:183)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:210)
at org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:312)
at org.apache.http.impl.execchain.ResponseEntityProxy.streamClosed(ResponseEntityProxy.java:142)
at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)
at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)
at java.io.BufferedInputStream.close(BufferedInputStream.java:483)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at org.glassfish.jersey.apache.connector.ApacheConnector...

Jersey 2.30 contains a workaround for the changed Apache Http Client 4.5.1+ and the timeout is not needed to be added in the client code anymore.

This entry was posted in Jersey. Bookmark the permalink.

5 Responses to Jersey Apache Connector Hangs …?

  1. Tejaswini says:

    Can we use Apache CXF libraries in Jersey appliation? For example, I want to use the FIQL search capability provided by cxf-rt-rs-extension-search jar in my Jersey application. When I include this dependency in my war deployment, I get “HK2 service reification failed” for the context classes included in the library.

    • Jan says:

      CXF is a JAX-RS implementation, such as Jersey. While both share the common JAX-RS API, the CXF and Jersey internals are not compatible, I am afraid. As far as I am aware, CXF does not use HK2 for the injection, so that might be the issue you experience.

  2. Jonathan says:

    Thanks for sharing the information!
    What is the motivation to move to ApacheConnectorProvider? HttpUrlConnectorProvider provides connection pooling as well (through the JVM’s implementation of HttpUrlConnector). Further more, Jersey documentation warns you that using the non-default provider might lead you to trouble. If you have some insights on that, I’d be thrilled to hear.

  3. Yev says:

    Great post.
    I wanted to ask, what are the reasons for moving from the default HttpUrlConnectorProvider to ApacheConnectorProvider ?

    • Jan says:

      Jersey supports multiple Connector Providers, each of which has some advantage over the default HttpUrlConnector, and also some disadvantages. Mostly, it is the broader feature set provided by each implementation that brings customers to the provider. In the case of Apache connector, the ApacheClientProperties file contains the set of supported features. Apache Http Client also does a good job when reusing http connections.

Leave a Reply to Tejaswini Cancel reply

Your email address will not be published. Required fields are marked *