Requesting objects from a static list that cause spikes in CPU utilization on the server

Question

We have a feature similar to related questions about SO where we show related records when viewing this record. Related records were retrieved by calling db each time the page was loaded. To reduce the load on db, I created a static list of these records that are loaded on Application_Start, and now I use the Linq query to query this list.

For the most part, this seems to work fine. The db load has decreased, and the Sql profiler shows that there are no more queries for related records. I went through this change yesterday, and this morning I found that the IIS workflow with the processor is 100% and the website is not responding. I switched to the old code (where I requested db) and everything was fine on the web server, but the db load increased. So I switched back to the new code and looked at the processor load on the web server. This is what I noticed.

When using the old code that db requests, the use of the IIS workflow processor is negligible and there are hardly any changes (a straight line is displayed on the performance tab of the task manager). However, when using new code that creates and queries a static list, I see spikes in the use of the IIS workflow processor. It’s basically from 1 to 5%, but from time to time it splashes over it, and the max that I saw is 40% since I started watching. I wonder why this is happening and if it is under heavy load, could this lead to a malfunction of the workflow?

Here are some of the code that requests a static list

if (validSearchLatLong && (usePincodeLatLong || distanceFilterLimit != distanceLimit)) { filteredRecords = StaticRecords.Where(job => LatLongDistance(centerLatitude, centerLongitude, job.lat, job.lon) <= distanceFilterLimit || (!string.IsNullOrEmpty(this.PreferredJobCity) ? job.city == this.PreferredJobCity : false)).ToList(); } else { filteredRecords = StaticRecords.Where(job => (this.PreferredJobCity != "" ? job.city == this.PreferredJobCity : (this.City != "" ? job.city == this.city : (state != "" ? job.state.Trim() == state.Trim() : false)))).ToList(); } if (RecordsearchFilter.JobCategories.Count > 0 && RecordsearchFilter.EnableFilter) { filteredRecords = filteredRecords.Where(job => this.RecordsearchFilter.JobCategories.Contains(job.JobCategoryClass)).ToList(); } else { filteredRecords = filteredRecords.Where(job => MatchJobCategories(job.JobCategory, (short)this.jobCategory.SqlId) > 0).ToList(); } 

The list supports multiple parallel readers, so StaticRecords should not be a blocking factor. And after that, I create filterRecords, which is a new filtered list that should be independent of other threads.

What could be the possible causes of spikes in the CPU, and could this lead to an earlier failure of the IIS workflow?

============= EDIT ===============

Now I know that the accident was not caused due to this code, and the other error. The static list method was stable and seemed to work well to achieve its goal.

But my question still remains, why are there spikes in the processor?

When I speak

The list supports multiple simultaneous readers, so StaticRecords should not be a blocking factor. And after that I create filterRecords which is a new filtered list that should be independent of other threads.

Am I mistaken?

Answer 1

SQL Server is usually very efficient at querying, especially when you have optimized indexes correctly.

Your new StaticRecords approach requires scanning in memory of all elements, executing the Where predicate for each. This is probably the reason for the increase in CPU usage, especially if there are many elements in StaticRecords . Try connecting the profiler and see for yourself. In general, I would expect that such filtering would work an order of magnitude better when working on SQL Server and filtering in memory.

Answer 2

Performance issues are probably related to List scans in StaticRecords .

Reading data into memory once is fine, but it seems that an efficient query requires a more efficient data structure. You can also return to the idea of ​​using LINQ to execute a query.

Regarding the causes of the crash and the 100% spike in the processor, I don't see an obvious error in the code you posted.