File descriptor error while receiving response code. (cxf, ssl)

Question

We have had a problem in production for quite some time ...

This is the next question: my other question , but with more detailed information, so I believe that posting it as a new question is justified (if not just add this information to another question).

Here he is:

So, we have a file descriptor leak in aix with (ibm) java 6 weblogic using an application using cxf, and we access our web service as well as jsb that goes to our ws.

When using File Leak Detector as an agent, when starting weblogic and resetting getCurrentOpenFiles () and filtering on Listener.SocketRecord programmatically, we have 2000+ open sockets;

These are java sockets and file descriptors, network sockets (viewed using netstat) are properly closed in time, but software (and those viewed using

lsof -p $pid_of_managed_server 2> /dev/null|grep TCP|wc -l 

) remain open (and ultimately cause too many problems with open files)).

This is the stack chapter of one of these open file descriptors inside jvm:

 record socket to tst-cjcsr.just.fgov.be/10.239.7.19:443 by thread:[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)' on Thu Nov 10 10:25:56 CET 2016 at java.net.PlainSocketImpl.create(PlainSocketImpl.java:188) at java.net.Socket.createImpl(Socket.java:411) at java.net.Socket.connect(Socket.java:544) at weblogic.net.http.HttpsClient.openWrappedSSLSocket(HttpsClient.java:565) at weblogic.net.http.HttpsClient.openServer(HttpsClient.java:296) at weblogic.net.http.HttpsClient.openServer(HttpsClient.java:373) at weblogic.net.http.HttpsClient.New(HttpsClient.java:528) at weblogic.net.http.HttpsURLConnection.connect(HttpsURLConnection.java:239) at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:409) at weblogic.net.http.SOAPHttpsURLConnection.getInputStream(SOAPHttpsURLConnection.java:37) at weblogic.net.http.HttpURLConnection.getResponseCode(HttpURLConnection.java:1038) at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:266) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.doProcessResponseCode(HTTPConduit.java:1550) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1579) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1520) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1317) at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:632) at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272) at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:572) at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:481) at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:382) at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:335) at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96) at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:136) at com.sun.proxy.$Proxy380.requestCriminalRecord(Unknown Source) at be.fgov.just.cjr.application.dossier.DossierBean$1.call(DossierBean.java:225) 

I can imagine that this is an applied problem, because I can not find other cases online with similar defects.

Can anyone understand more from this stack?

For example: it seems strange to me that HTTPConduit.close () wants to create a connection ...

Another point: the problem does not occur when using the same set of technologies for calls other than https. (which makes sense because stacktrace mentions the Http s client)

CXF Version: 2.7.18

cxf config:

 <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:jaxws="http://cxf.apache.org/jaxws" xmlns:soap="http://cxf.apache.org/bindings/soap" xmlns:cxf="http://cxf.apache.org/core" xmlns:context="http://www.springframework.org/schema/context" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd http://cxf.apache.org/core http://cxf.apache.org/schemas/core.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.1.xsd http://cxf.apache.org/jaxws http://cxf.apache.org/schemas/jaxws.xsd"> <import resource="classpath:META-INF/cxf/cxf.xml"/> <context:property-placeholder location="file:${config.file.location}cjr-extract-${cjr.environment}.properties"/> <jaxws:client id="CJCSCGService" serviceClass="be.fgov.just.cjr.ws.extract.generated.CJCSCGService" address="${ws.extract.wsdl.endpoint}"> <jaxws:binding> <soap:soapBinding version="1.2"/> </jaxws:binding> </jaxws:client> <cxf:bus> <cxf:outInterceptors> <bean class="our.package.interceptors.SomeInterceptor" id="webSecurityInterceptor"> <constructor-arg> <map> <entry key="action" value="Timestamp Signature"/> <entry key="user" value="${org.apache.ws.security.crypto.merlin.keystore.alias}"/> <entry key="passwordCallbackRef"> <ref bean="passwordCallBack"/> </entry> <!--entry key="signaturePropFile" value="properties/our.properties"/--> <entry key="signaturePropFile" value="file:${location}/our.properties"/> <entry key="signatureKeyIdentifier" value="DirectReference" /> </map> </constructor-arg> </bean> </cxf:outInterceptors> <cxf:properties> <entry key="signatureParts" value="{Element}{http://www.w3.org/2003/05/soap-envelope}Body;{Element}{http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd}Timestamp"/> </cxf:properties> </cxf:bus> <bean id="passwordCallBack" class="our.package.authentication.PasswordCallbackHandler"> <property name="password" value="${password}"/> <property name="alias" value="${org.apache.ws.security.crypto.merlin.keystore.alias}"/> </bean> </beans> 

Java:

 java version "1.6.0" Java(TM) SE Runtime Environment (build pap3260sr15fp1-20140110_01(SR15 FP1)) IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 AIX ppc-32 jvmap3260sr15- 20131231_180656 (JIT enabled, AOT enabled) J9VM - 20131231_180656 JIT - r9_20130920_46510ifx3 GC - GA24_Java6_SR15_20131231_1152_B180656) JCL - 20140107_01 

Weblogic 10.3.6

Thanks for the tips, pointers and, of course, - if possible - the answers; -)

S.

The requested EJP code, here it is:

The call itself:

 private CJCSCGService cjcscgService; private Callable<RequestCriminalRecordResponse> callService(final RequestCriminalRecordRequest request) { return new Callable<RequestCriminalRecordResponse>() { @Override public RequestCriminalRecordResponse call() throws Exception { try { return cjcscgService.requestCriminalRecord(request); // this is line 225. } catch (Exception e) { facesMessages.error("technicalError"); log.error("Encountered technical error", e); return null; } } }; } 

Web service generated:

 package be.fgov.just.cjr.ws.extract.generated; import javax.jws.WebMethod; import javax.jws.WebParam; import javax.jws.WebResult; import javax.jws.WebService; import javax.jws.soap.SOAPBinding; import javax.xml.bind.annotation.XmlSeeAlso; /** * This class was generated by the JAX-WS RI. * JAX-WS RI 2.1.7-b01- * Generated source version: 2.1 * */ @WebService(name = "CJCSCGService", targetNamespace = "http://secret/service-v1.0-rc2") @SOAPBinding(parameterStyle = SOAPBinding.ParameterStyle.BARE) @XmlSeeAlso({ ObjectFactory.class }) public interface CJCSCGService { /** * * @param requestCriminalRecordRequest * @return * returns be.fgov.just.cjr.ws.extract.generated.RequestCriminalRecordResponse */ @WebMethod(action = "http://secret/service/RequestCriminalRecord") @WebResult(name = "requestCriminalRecordResponse", targetNamespace = "http://secret/service-v1.0-rc2", partName = "requestCriminalRecordResponse") public RequestCriminalRecordResponse requestCriminalRecord( @WebParam(name = "requestCriminalRecordRequest", targetNamespace = "http://secret/service-v1.0-rc2", partName = "requestCriminalRecordRequest") RequestCriminalRecordRequest requestCriminalRecordRequest); } 

from:

 <http-conf:conduit name="*.http-conduit"> <http-conf:client Connection="Close" /> </http-conf:conduit> 

Connections (os level) close; but we still have a leak (the behavior is indicated above).

from:

 <http-conf:conduit name="*.http-conduit"> <http-conf:client Connection="Keep-Alive" /> </http-conf:conduit> 

Both connections (os level) as file descriptors continue to grow ...

(additional information: os level connections go down (after a timeout, I think), but file descriptors remain open ...)

Answer 1

From my point of view, the problem is with the cxf configuration, maybe interacting with weblogic + cxf is also a problem, because it is the cxf component that controls the connection (see stack trace). Therefore, I suggested trying with various keep-alive settings (how to install keep-alive is described here ). It didn't seem to help break open file descriptors.

UPDATE:

Think more about your problem:

 at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:266) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.doProcessResponseCode(HTTPConduit.java:1550) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1579) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1520) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1317) 

it seems that when closing the http connection, cxf tries to return a response code and creates a socket for it, and since these sockets just wait for a response that never comes on indefinitely, which creates a leak. The entire flow is described here in the simplified client workflow section. So the next question is: does tst-cjcsr.just.fgov.be ? And my next suggestion is to configure ReceiveTimeout for http-conf:client , as described above.