Dears
we are faced with the problem "too many open files".
We have made some progress, but the problem still exists ...
We use cxf in weblogic on aix.
Corresponding cxf configuration:
<http-conf:conduit name="*.http-conduit"> <http-conf:client Connection="close" /> </http-conf:conduit>
When executing this code:
for(int i=0;i<1300;i++){ response = callService(request).call(); }
(we do not intend to use this code in the production process, but it should fix the problem)
-> the first launch is performed;
-> second run provides problem
output:
netstat| wc -l
first showed that too many sockets remain open (hence cxf configuration change)
now open sockets remain stable + - 400 lines, where until configuration change> 1000
but...
lsof 2> /dev/null|grep TCP|wc -l
reveals the problem that we are currently facing
the number of open sockets is stable (therefore, sockets are closed), but the number of open files is not ...
from 319 immediately after loading the managed server to +1688 when the following exception occurs:
javax.xml.ws.soap.SOAPFaultException: NullPointerException invoking https://tst-cjcsr.just.fgov.be/cjcs-cg-ws/1.0-rc2/: null at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:158) at com.sun.proxy.$Proxy302.requestCriminalRecord(Unknown Source) at be.fgov.just.cjr.application.dossier.DossierBean$1.call(DossierBean.java:226) at be.fgov.just.cjr.application.dossier.DossierBean$1.call(DossierBean.java:222) at be.fgov.just.cjr.application.dossier.DossierBean.callService(DossierBean.java:153) at be.fgov.just.cjr.application.dossier.DossierBean.search(DossierBean.java:99) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:329) at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:342) at org.jboss.el.parser.AstPropertySuffix.invoke(AstPropertySuffix.java:58) at org.jboss.el.parser.AstValue.invoke(AstValue.java:96) at org.jboss.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:276) at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:105) at javax.faces.component.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:88) at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102) at javax.faces.component.UICommand.broadcast(UICommand.java:315) at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:794) at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1259) at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:81) at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101) at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:409) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227) at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301) at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at be.fgov.just.cjr.application.authentication.UserbeanEnforceFilter.doFilter(UserbeanEnforceFilter.java:64) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at be.fgov.just.cjr.application.authentication.openam.OpenAmPingFilter.doFilter(OpenAmPingFilter.java:88) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at be.fgov.just.cjr.application.authentication.openam.OpenAmReverifyFilter.doFilter(OpenAmReverifyFilter.java:94) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at be.fgov.just.cjr.application.authentication.openam.OpenAmSessionFilter.doFilter(OpenAmSessionFilter.java:166) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60) at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3748) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3714) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2283) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2182) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1491) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263) at weblogic.work.ExecuteThread.run(ExecuteThread.java:221) Caused by: java.lang.NullPointerException: NullPointerException invoking https://tst-cjcsr.just.fgov.be/cjcs-cg-ws/1.0-rc2/: null at sun.reflect.GeneratedConstructorAccessor108.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:39) at java.lang.reflect.Constructor.newInstance(Constructor.java:527) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1346) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1335) at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:632) at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272) at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:572) at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:481) at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:382) at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:335) at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96) at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:136) ... 56 more Caused by: java.lang.NullPointerException at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:459) at weblogic.net.http.SOAPHttpsURLConnection.getInputStream(SOAPHttpsURLConnection.java:37) at weblogic.net.http.HttpURLConnection.getResponseCode(HttpURLConnection.java:1038) at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:266) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.doProcessResponseCode(HTTPConduit.java:1550) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1579) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1520) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1317) ... 66 more
if additional information is required, just ask
any help in fixing this problem is much appreciated
restarting the managed server "solves" the problem, but this is not an option in production ...
TIME_WAIT is set to 1 (15 seconds), so there is no solution ...
S.
change
described one partial solution in Zone ; actually:
ulimit -S -n
outputs 2000 and
procfiles 15728790 | grep rlimit
exits
Current rlimit: 2000 file descriptors
Now we are implementing their solution (add ulimit -S -n 65536 to the end of resetFd ()), but the root cause (socket leak) is still not resolved: - /
Any help is still much appreciated ...
S.