Blocking convoys on an ASP.NET website

Question

TL DR: Made refactoring for performance, the site has become slower. Design a Concurrency Visualizer, the chart looks like blocking convoys , as described in MSDN.

Context

Im helps refactoring an ASP.NET website to switch user controls to execute business logic on datasets to execute business object presentation logic, as well as reduce database queries executed from user controls.

Problem

We noticed a significant decrease in performance (freezes / locks) after making changes related to what, in our opinion, will be an improvement in performance in several areas.

Use Lean Sentry to monitor the performance of our websites. According to the hangup diagnostics, there were no threads in the thread pool and (as described in the diagnostics page), when the GC is running, this stops the creation of new threads. According to the memory diagnostics, GC Heap and Gen 0 consumed a lot of memory (~ 9 GB).

What have i done so far?

• I used the memory profiler in Visual Studio and identified problems with our excessive use of the DataAdapter and DataTable . Memory consumption dropped to 3 GB, but it only helped with locking the GC. It is still slower than before we made the changes, and I still see the lock under high load caused by functions like CompilationLock.GetLock() and BuildManager.GetBuildResultFromCacheInternal() . Googling them did not return anything useful.

• This is a website that uses JIT compilation. I suggested that the CompilationLock problem might be related to JIT compilation and wants to run a precompiled website , but one of our global Utilities classes has caused ambiguity with some other Utilities class / namespace that I don't know about. Ive learned that there is a Microsoft.Build.Utilities namespace, but it is not mentioned on our website, and I cannot reproduce ambiguity in my own environment when I refer to Microsoft.Build on my own, so I could not get the site to work in precompiled mode at the server stage to test this theory.

• I made additional changes in memory allocation and the number of calls in the database using Visual Studios memory allocation and toolkit profilers , but I did not notice any performance progress.

• I used the concurrency profiler to collect additional information about using threads. I have not used this tool before, so I'm not sure about my interpretations here. There are several threads in each descriptor, and in one hand Im sees 42% of the competition. I see that the methods DataAdapter.Fill and SqlHelper.ExecuteReader appear most often when its set "Show only my code" and WaitForSingleObjectExImplementation displayed most often when its set "Show all code".

• I ran into the SO question ASP.NET website performance issues and set EnableSessionState="ReadOnly" for each page , but I also did not notice the difference with this change.

• concurrency Visualizer and General Patterns for Poorly Managed Multithreaded Applications helped me identify the problem. My schedule does not look like a batch, but I see synchronization at 80-90%, as shown in the Lock Convoys chart. I checked the SO question for debugging lock convoys .

Testing Approach

I use Screaming Frog to scan a website to reproduce problems and requests per second and response time in Screaming Frog and Lean Sentry as an indicator of performance. This may not be the best way, but the difference is noticeable, reproducible and almost everything that I have at the moment.

Website Architecture

The website was originally encoded in VB.NET for the .NET Framework 1.0 about 10 years ago and updated to the .NET Framework 4.6.1, fixing some compatibility issues. Until now, there have been no architectural changes. There is a common SqlHelper class, which is a set of common data access functions, such as ExecuteDataset or ExecuteDatareader , that return either a DataSet , DataReader , or String . These functions read connection string information from the web.config and create new SqlConnection , SqlDataAdapter , SqlDataReader and SqlCommand objects to perform database operations. The data access level that this generic class uses consists of classes for each module, such as a shopping cart, category, product, etc. Which must be created in each user control and consist of functions that represent stored procedures in the database.

Refactoring

We have introduced some new objects that must be installed either inside the page load of the associated user control or inside the OnItemDataBound event of the relays and are tied to its child user, which control the public properties that are reorganized to use the object. However, there are other controls for child users who need several data tables, so we decided to save one of the data tables in one of the objects and pass it to the associated user controls, assigning them to our public properties.

I assume that we damaged performance by introducing these objects. Despite the fact that database calls and memory consumption seem to be reduced, Im wonders if the objects are causing the threads to constantly synchronize threads.

Graph before any refactoring:

The chart after all the refactoring that I talked about is applicable:

Help me identify the problem?

Answer 1

Your problem is quite complicated. I think that you have two main options for solving your refactoring problems:

• Revert code changes until all or a lot of the refactoring has been completed, and when you get better performance than what you are currently experiencing. Then, gradually, add new classes to improve performance. If the change does not improve performance, undo it and try something else.

• Replace some / many recently added classes with versions that support interfaces but have no performance overhead. Do this selectively to isolate where performance problems exist. The website may have detected an unknown performance error that was not caused by previous implementations of the added classes.

I would prefer option 1, although this may seem counterproductive. It is a bit like football in the USA. Of course, it's nice to just go down the field. But sometimes the dominant strategy is to punt, get the ball back and try to score another drive.