Over the past few days, the team has struggled with a constant problem when, at very predictable intervals, one instance of ColdFusion was a white screen of death.
Every three hours, the site will simply start returning a blank white page for any URL. Then we restarted the instance, and everything would be great ... another three hours, almost up to a minute. Of course, this happened on Friday, so that every weekend people took turns reloading the instance each time it died.
As far as I can tell, no one made any changes to either ColdFusion or our server environment until it started. Prior to this, the instance worked fine.
Since then, we have seen that the isapi_redirect.log file for this instance is filled with Tomcat / connection errors.
We followed the excellent instructions at http://www.webtrenches.com/post.cfm/resolve-stability-problems-and-speed-up-coldfusion-10 and adjusted our connector settings as recommended. Although this may have helped the overall performance very well and changed the timeframe from 3 to 3.5 hours between failures, it did not resolve it.
Before that, we even tried to move the site from one of our virtual servers to another without any luck.
We tried to restart IIS and even restart the whole server in one night to see if that would help, and still nothing.
, , . , , , , , .
IIS v7.5.7600.16385
/IIS, , , " -".
, , - ... IIS , , http.log .
, tomcat .
, , CF, , .
CF ( ).
, , CF , , .
Server Product: ColdFusion
Version: 10,0,13,287689
Tomcat Version: 7.0.23.0
Edition: Enterprise
Operating System: Windows Server 2008 R2
OS Version: 6.1
Update Level: chf10000013.jar
Adobe Driver Version: 4.1 (Build 0001)
workers.properties:
worker.list=Instance_Codebase
worker.Instance_Codebase.type=ajp13
worker.Instance_Codebase.host=localhost
worker.Instance_Codebase.port=8014
worker.Instance_Codebase.max_reuse_connections=250
worker.Instance_Codebase.connection_pool_size=250
worker.Instance_Codebase.connection_pool_timeout=60
server.xml
<Server port="8009" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on"></Listener>
<Listener className="org.apache.catalina.core.JasperListener"></Listener>
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"></Listener>
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"></Listener>
<GlobalNamingResources>
<Resource description="User database that can be updated and saved" name="UserDatabase" pathname="conf/tomcat-users.xml" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" type="org.apache.catalina.UserDatabase" auth="Container"></Resource>
</GlobalNamingResources>
<Service name="Catalina">
<Executor name="tomcatThreadPool" minSpareThreads="4" maxThreads="150" namePrefix="catalina-exec-"></Executor>
<Connector port="8014" protocol="AJP/1.3" redirectPort="8447" tomcatAuthentication="false" maxThreads="250" connectionTimeout="60000"></Connector>
<Engine jvmRoute="Instance_Codebase" name="Catalina" defaultHost="localhost">
<Realm className="org.apache.catalina.realm.LockOutRealm">
<Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"></Realm>
</Realm>
<Host name="localhost" autoDeploy="false" unpackWARs="true" appBase="webapps">
</Host>
</Engine>
<Connector port="8501" protocol="org.apache.coyote.http11.Http11NioProtocol" connectionTimeout="20000" redirectPort="8443" executor="tomcatThreadPool"></Connector>
</Service>
</Server>
isapi_redirect.log. http://trasper.com/files/isapi_redirect.log.txt.
( ) 23:41, .
[Wed Jun 25 23:40:34.503 2014] [10012:912] [info] ajp_send_request::jk_ajp_common.c (1658): (Instance_Codebase) all endpoints are disconnected, detected by connect check (27), cping (0), send (0)
[Wed Jun 25 23:40:34.504 2014] [10012:1396] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1313): (Instance_Codebase) can't receive the response header message from tomcat, network problems or tomcat (127.0.0.1:8014) is down (errno=54)
[Wed Jun 25 23:40:34.820 2014] [10012:1396] [error] ajp_get_reply::jk_ajp_common.c (2190): (Instance_Codebase) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Jun 25 23:40:34.823 2014] [10012:1396] [info] ajp_service::jk_ajp_common.c (2692): (Instance_Codebase) sending request to tomcat failed (recoverable), (attempt=1)
[Wed Jun 25 23:40:34.708 2014] [10012:7880] [error] ajp_get_reply::jk_ajp_common.c (2190): (Instance_Codebase) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Jun 25 23:40:40.477 2014] [10012:2296] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1047): Failed opening socket to (127.0.0.1:8014) (errno=61)
[Wed Jun 25 23:40:40.364 2014] [10012:8256] [error] ajp_service::jk_ajp_common.c (2711): (Instance_Codebase) connecting to tomcat failed.
[Wed Jun 25 23:40:40.825 2014] [10012:7060] [error] HttpExtensionProc::jk_isapi_plugin.c (2309): service() failed with http error 503
[Wed Jun 25 23:40:40.877 2014] [10012:10364] [error] ajp_send_request::jk_ajp_common.c (1669): (Instance_Codebase) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=61)
[Wed Jun 25 23:40:40.965 2014] [10012:10364] [info] ajp_service::jk_ajp_common.c (2692): (Instance_Codebase) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)
[Wed Jun 25 23:40:40.857 2014] [10012:1020] [error] HttpExtensionProc::jk_isapi_plugin.c (2309): service() failed with http error 503