I am trying to figure out bottlenecks in a web application on a production server. ColdFusion 10 runs about 80 sites with the latest versions of Java 8, Tomcat 7, IIS 8.5 on Windows Server 2012 R2. Random requests peak from an average of 500-1000 ms to 30 seconds, which means that 9 out of 10 requests of the same page are completed within 1000 ms, and one takes 20-30 seconds. It seems that this can happen on almost every page / template with code.
Web Application Features:
- many regular expressions and string manipulations
- many data transformations (list for array, recreation of arrays, ordering of data order, creation of structures)
- lightweight SQL queries (dedicated SQL server on the same network)
- no outgoing connections (
<cfhttp>)
I am tracking the use of FusionReactor, and I noticed the following things:
- the server has very low traffic (an average of 6 requests per second, 50 at best).
- requests are not queued
- the average is 60 JDBC requests per second between 1-2 ms, the highest is 250 JDBC requests per second, the worst total time is about 300 ms
- the heap at best is 10 GB (most of them are cache / old gene, space for eden is about 2 GB, space for survivors is less than 200 MB)
- GC young collection starts every 40 seconds in about 20 ms on average, the worst time is 140 ms
- disk read / read is on average less than 200 Kbps, maximum read speed is 2 Mbps, maximum write is 4 Mbps
- IO network: 200 bit / s TX, 400 b / s RX average
- CPU runs from 20% to 60% of the load
- , 20-30 , JIT Time
Stacktrace
. . . , , , , <cfinclude>, .
2500 , - 29727 . TTFB TTLB 29726 , 2 1 . - 76 (19 ).
, funcCONSTRUCTFOOTER.getAccess - Java- return 0; ( access <cffunction>), , 23 . /, .
:
JVM (Java HotSpot(TM) 64-Bit Server VM, 1.8.0_121)
-server
-Xms24G
-Xmx48G
-Xss4m
-XX:MaxMetaspaceSize=2G
-XX:+TieredCompilation
-XX:ReservedCodeCacheSize=2G
-XX:+UseCompressedOops
-XX:+UseG1GC
-XX:MaxGCPauseMillis=100
-Xbatch
ColdFusion (10.0.22.283922, 22)
Maximum number of simultaneous Template requests: 40
Maximum number of cached templates: 40000 (~14500 cached according to monitor)
Trusted cache: true
Cache template in request: true
Component cache: true
Save class files: true
<Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="500" connectionTimeout="60000" />
worker.list=cfusion
worker.cfusion.type=ajp13
worker.cfusion.host=localhost
worker.cfusion.port=8012
worker.cfusion.connection_pool_size=500
worker.cfusion.max_reuse_connections=250
worker.cfusion.connection_pool_timeout=60
metrics.log 35 75 . 90% , . Max threads: 500 Current thread count: 70 Current thread busy: 64.
? ?