I have an unpleasant problem that I find it difficult to explain. Simply put, CPU usage is inexplicably high on the web servers in my web farm.
I have a large number of users who click on two front-end web servers. 99% of page loads are Ajax requests and serve as a simple serialized JSON object that web servers retrieve from the backend using WCF. In a typical case (again, probably 99% of the requests), the entire ASPX page makes a WCF call to retrieve this data, serialize it into a JSON string, and return it.
The object is quite small - a pointer, a couple of short lines, a few ints.
A non-standard case is the initial loading of a page that does the same (WCF request), but enters the response into different parts of the page using asp: literals.
All three machines (2 web servers, one backend) have the same rigid specification. I would expect the backend to do most of the work in this situation, as it manages all the data, performs a search, etc. BUT: loading on the backend is much less than the load on the front ends. The backend is a good 10-20% load. The front ends have an average of 30%, but they are all over the map, sometimes hitting 100% peaks for 10 seconds and taking 600 ms to serve these very simple pages.
When I run the front-end in the profiler (ANTS), it associates a WCF connection with 80% of the processor time. That the whole call is a .NET-generated WCF proxy.
Configuring WCF: The service is completely parallel. I have set single mode and concurrency set to multiple. I opened maxConnections and listenBacklog in a service up to 256. With a lot of stress (500 requests / I), I see about 75 connections open between both servers and the service, so it does not fall on this wall. I have a no protection. The bandwidth usage is about 1/20 of the potential (4 Mbit / s on a 100 Mbit / s network).
On the client (web servers), I create a static ChannelFactory for the service. The code to call the service is as follows:
service = MyChannelFactory.CreateChannel(); try { service.Call(); service.Close(); } catch { service.Abort(); }
(simplified, but you get the basic picture)
What I don’t understand is where all this load on the front end comes from. What is strange is that it is never in the range of 30% -90%. This is either in a panic mode (100%) or in order (30% or less). However, given the server load, I expected both of these machines to be 10% or less. Using memory, pens, etc. Seem reasonable.
To add another wrinkle: when I register how long it takes to service these calls on the backend, I get the time in succession less than 15 ms (maybe one or two peaks up to 30 ms every minute). At the front end, these calls can take up to 1 s. I think this may be due to problems with the CPU, but it seems to me that this is not so.
So ... does anyone have any ideas on where to look for such things? I quickly figure out things to explore.
Clarification . The WCF service is hosted on a Windows service and uses netTcp binding. In addition, I have maxConnections on a client set to 128, FWIW.