All geek questions in one place

Weblogic and Oracle: Stuck ExecuteThread

We run SpringMVC with Hibernate in a Weblogic clustered environment. Our application worked well until we ran into database space issues. In this regard, our database provider has expanded the database storage.

After that, the Weblogic system state change complains about the thread pool "Thread pool started threads".

The stack trace in the log file is as follows:

<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '12' for queue: 'weblogic.ke rnel.Default (self-tuning)' has been busy for "641" seconds working on the request "weblogic.servlet.internal.ServletReq uestImpl@17782ff3 [ GET /default/default/applicationPreloader/load.do?_=1331887527968 HTTP/1.1 User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:10.0.2) Gecko/20100101 Firefox/10.0.2 Accept: */* Accept-Language: en-gb,en;q=0.5 Accept-Encoding: gzip, deflate x-requested-with: XMLHttpRequest Referer: http://novintegweb01.eu.acme.net:9136/default/default/applicationPreloader.do Cookie: JSESSIONID1=NhBdPv9Hx8Wj5lnr5vnGhLSbR2GGM1DPhqT2YGzWCGFvhN3WJWVH!700169023!1391220923 Connection: Keep-Alive Proxy-Client-IP: 192.168.1.1 X-Forwarded-For: 192.168.1.1 X-WebLogic-KeepAliveSecs: 30 X-WebLogic-Request-ClusterInfo: true x-weblogic-cluster-hash: G4kGT1kfuDBbtj5w//SYNt6hlVo ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace: java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.read(SocketInputStream.java:129) oracle.net.ns.Packet.receive(Packet.java:293) oracle.net.ns.DataPacket.receive(DataPacket.java:104) oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315) oracle.net.ns.NetInputStream.read(NetInputStream.java:260) oracle.net.ns.NetInputStream.read(NetInputStream.java:185) oracle.net.ns.NetInputStream.read(NetInputStream.java:102) oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124) oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80) oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1136) oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1113) oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288) oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191) oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523) oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207) oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1009) oracle.jdbc.driver.OraclePreparedStatement.executeForRowsWithTimeout(OraclePreparedStatement.java:10117) oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:10221) oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:216) weblogic.jdbc.wrapper.PreparedStatement.executeBatch(PreparedStatement.java:210) org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:48) org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:247) org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:93) org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:88) org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:223) org.hibernate.persister.collection.AbstractCollectionPersister.recreate(AbstractCollectionPersister.java:1119) org.hibernate.action.CollectionRecreateAction.execute(CollectionRecreateAction.java:39) org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279) org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263) org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:171) org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298) org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27) org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1001) org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:339) org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106) org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:656) org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754) org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723) org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393) org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:120) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621) com.acme.superproject.dao.CertificationDaoImpl$$EnhancerByCGLIB$$bf3135a5.getById(<generated>) com.acme.superproject.integration.UserProfileSabaServiceImpl.determineTargetDateFromWs(UserProfileSabaServiceImpl.java:436) com.acme.superproject.integration.UserProfileSabaServiceImpl.determineTargetDate(UserProfileSabaServiceImpl.java:401) com.acme.superproject.integration.UserProfileSabaServiceImpl.createCertificationRelationObject(UserProfileSabaServiceImpl.java:373) com.acme.superproject.integration.UserProfileSabaServiceImpl.getRequiredLearning(UserProfileSabaServiceImpl.java:334) com.acme.superproject.integration.UserProfileSabaServiceImpl.extractUserProfile(UserProfileSabaServiceImpl.java:203) com.acme.superproject.integration.UserProfileSabaServiceImpl.getUserProfile(UserProfileSabaServiceImpl.java:73) com.acme.superproject.integration.UserProfileSabaServiceImpl.getUserProfile(UserProfileSabaServiceImpl.java:114) com.acme.superproject.integration.UserProfileSabaServiceImpl$$FastClassByCGLIB$$e48c61c9.invoke(<generated>) net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149) org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) com.acme.superproject.techarch.TracingAspect.trace(TracingAspect.java:31) sun.reflect.GeneratedMethodAccessor5382.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:597) org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621) com.acme.superproject.integration.UserProfileSabaServiceImpl$$EnhancerByCGLIB$$5d7b85df.getUserProfile(<generated>) com.acme.superproject.service.UserProfileServiceImpl.getUserProfile(UserProfileServiceImpl.java:26) com.acme.superproject.service.UserProfileServiceImpl$$FastClassByCGLIB$$70cfe815.invoke(<generated>) net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149) org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) com.acme.superproject.techarch.TracingAspect.trace(TracingAspect.java:31) sun.reflect.GeneratedMethodAccessor5382.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:597) org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621) com.acme.superproject.service.UserProfileServiceImpl$$EnhancerByCGLIB$$5bcb212b.getUserProfile(<generated>) com.acme.superproject.web.session.SessionDataProvider.fetchUserProfile(SessionDataProvider.java:97) com.acme.superproject.web.session.SessionDataProvider.getUserProfile(SessionDataProvider.java:78) com.acme.superproject.web.session.SessionDataProvider.fetchUserRegistrationCollection(SessionDataProvider.java:155) com.acme.superproject.web.session.SessionDataProvider.getUserRegistrationCollection(SessionDataProvider.java:141) com.acme.superproject.web.ApplicationPreloaderController.loadAction(ApplicationPreloaderController.java:84) sun.reflect.GeneratedMethodAccessor6506.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:597) org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176) org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:426) org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:414) org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790) org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719) org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644) org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549) javax.servlet.http.HttpServlet.service(HttpServlet.java:707) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227) weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125) weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300) weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:368) org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109) org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380) org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380) org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380) org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:89) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380) org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380) org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:169) org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715) weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681) weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277) weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183) weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454) weblogic.work.ExecuteThread.execute(ExecuteThread.java:207) weblogic.work.ExecuteThread.run(ExecuteThread.java:176) >

We have already restarted Weblogic, the database server, and made a new deployment. The problem is always the same: when we try to access the login page of the application, it freezes - as far as I can see when trying to execute a database transaction.

In addition to this, when I try to connect to our database using "Oracle SQL Developer", we also get the error "ORA-00257: archiver error. Only connect internal ones before release." Unfortunately, we do not have SYSDBA rights (because they are managed by an external provider).

My question is:

  • Have any of you had a similar problem before?
  • Do you think that the main reason for such problems is more on the application side or on the database side?
  • How can I at least get rid of stuck threads?
+6
source share
3 answers

Most likely, the database store is full or that the archiver cannot move or create new archive logs. This is almost certainly a problem at the database level, not at the application server level. Since you receive this message, it means that you already have a connection to the database, so it is not sitting and is not waiting for a connection.

Stuck streams in WebLogic do not always have to be fixed and are not always a problem. The purpose of the stuck thread mechanism is to notify you when a thread takes an abnormally long time to do something (for example, access a database). A recorded stream notification is only a notification that something may be wrong, in which case it serves this purpose.

If you have threads that usually take 20 minutes to do something, but want to get a notification if you need more than 25 minutes, then you need to increase the Stuck Thread Limit to 25 minutes and get a notification at this point, then you should put this limit is from 600 seconds to 1500 seconds (25 * 60).

You can edit the stuck thread limit in weblogic admin base_domain> Environment> Servers> Admin Server> Configuration / Setup.

see link below: https://docs.oracle.com/cd/E19316-01/820-3745/ghudn/index.html

+2
source

This seems to be a database problem. Threads are stuck in the database. Your productivity seems to have gotten worse. I would suggest referring the matter to your database administrator for further investigation. Also, try to recreate your table access indexes.

0
source

We encountered this problem when the Oracle DB Server was in VirtualBox in the development environment. Indication of the problem was difficult. We noticed this when ThreadPool had no more threads to process requests, and requests were rejected.

0
source

Source: https://habr.com/ru/post/910897/

More articles:


All Articles