How to debug ASP.NET popup process memory dump?

Question

Sorry, I could not find a good way to talk about my real question.

I am running an ASP.NET website with high traffic on a 64 bit machine. However, IIS runs in 32-bit mode due to some legacy application components. I run this particular web application in the application pool, which has a web garden option (6 processes are running inside an 8-processor machine).

Once or twice a week, one of the processes will take off in 100% CPU usage, which will lead to a giant slowdown for the site, so my plan was to wait until this happens, the memory would throw offensive process, and then push WinDbg to zero in the stream that appears to see where the code rotates its wheels.

I debugged the use of WinDbg before figuring out what created the deadlock on the site, but that was a few months ago, and I don’t remember how I got it working. (As a note, this is a lesson to document everything you do.)

I run WinDbg on a server running Windows 2003 that runs this site to prevent problems with the version of the DLL. Here are my steps so far, please let me know where I am going wrong to receive the error message I receive.

• The first memory unloads the spiking process using UserDump with the following command, where 3389 is the process identifier:

  userdump -k 3389

• I am loading a dump into the x86 version of WinDbg.

• Since I run the 32-bit version on a 64-bit machine, I load the memory dump first, and then:

  .load wow64exts

  .effmach x86

• I am sure my symbol path includes a directory containing PDB files for applications:

  .sympath+ c:\inetpub\myapp\bin

• Only ".load SOS" is executed with the error "The system cannot find the specified file", so I go to the fully qualified route of the next action, which works:

  .load c:\windows\microsoft.net\framework\v2.0.50727\sos

From here I am lost. I try to execute any of the SOS commands, for example !threads , only to get this error:

 Failed to load data access DLL, 0x80004005 

This error is also accompanied by a numbered list of items that I should check. I verified that I am using the latest version of the debugger, mscordacwks.dll is actually located in the same directory as the mscorwks.dll file, and I am debugging the same architecture as the dump file.

I also run the magic command .cordll -ve -u -l , but that doesn’t solve anything. I always welcome " CLR DLL status: No load attempts " when I execute this. Then I try " .reload ", which gives some warnings like " WARNING: wldap32 overlaps dnsapi ". I would like to say something like " CLRDLL: Loaded DLL C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll ". But this is not so.

Answer 1

Try it! sw before running sos commands. See the blog post post .

Answer 2

• (MSDN) How to Use the CLR Profiler
• (MSDN Journal) Investigating Memory Problems

Answer 3

In my experience, a spike app pool can be caused by being recycled. Have you tried IIS Crash / Hang Agent and IMS Dump?

http://www.microsoft.com/downloads/details.aspx?FamilyID=01c4f89d-cc68-42ba-98d2-0c580437efcf&DisplayLang=en

Also bundled with them is the dumpfile analyzer, which will tell you about a memory leak and even suggest areas of your code that need to be fixed (complete with links to relevant MSKB articles!)

Answer 4

Dude - not sure if this helps, but maybe try this.

• Copy c: \ windows \ microsoft.net \ framework \ v2.0.50727 \ sos.dll to the same directory where windbg is installed (for example, c: \ program files \ Debugging Tools for Windows \). What for? simplify sos file loading.
• run windbg
• load the memory dump file. for me i use ctrl-D or File -> open crash dump
• .load sos <- take into account the full launch before the boot command
• .symfix c: \ temp \ debug_symbols
• .reload

Good .. pay attention to the command line. this tells me the current thread that the dump was. This may not be useful for the HIGH CPU scenario .. because we could be in any thread.

so from here I look at running threads and check the most loaded thread

eight! threadpool <- this means that I can see the processor being used to check that we are in a state of crap (busy) ... for example, 100% processor or not.

nine! runaway <is a list of threads that have ben around the longest ... for example.

 0:027 !runaway User Mode Time Thread Time 18:704 0 days 0:00:17.843 <-- Thread #18 19:9f4 0 days 0:00:13.328 <-- Thread #19 16:1948 0 days 0:00:10.718 26:a7c 0 days 0:00:01.375 24:114 0 days 0:00:01.093 27:d54 0 days 0:00:00.390 28:1b70 0 days 0:00:00.328 0:b7c 0 days 0:00:00.171 25:3f8 0 days 0:00:00.000 23:1968 0 days 0:00:00.000 

thread 18 and 19 hung around for a while .. hmm .... are they stuck in a loop?

10. ~ 18s <- goto thread 18.
11. ! clrstack <- clr call stack .., which is similar to debugging in windows.

.. and from here u can dump objects and stuff, indicating addresses and stuff.

check! help list some commands to try using .. i think! help.sos also work?

Hhh .. if you're still stuck, ask what worked and what didn't.

Answer 5

I just had to deal with a similar problem. In my case, it turned out that WinDbg could not find the correct version of mscorwks.dll. In addition to the Framework version, there is also a version of the DLL that may differ from the same version of the framework.

Theoretically, Microsoft character servers should be able to supply the necessary DLLs, but this was not for me. To solve this problem, I used !sym noisy for more information on loading characters. When I did !dumpstack , I received an error message:

SYMSRV: http://msdl.microsoft.com/download/symbols/mscorwks.dll/492B82C1590000/mscorwks.dll not found

To fix this, I created the appropriate folders in my local character cache and copied the mscorwks.dll file from the machine where the dump came from. After .reload WinDbg discovered the necessary DLL in the local symbol cache and continued happily.

Alternatively, you can find the exact version of mscorwks used with lm vm mscorwks . Then you can find the update containing the version you need for this list . You will need to extract the necessary DLLs from the specific update to the right place.