How to list running tasks in a .net memory dump

Question

We have a sophisticated ASP.Net Core application using the async / await template. The application has stopped responding recently, and we took a memory dump for it. We suspect that there is some asynchronous operation that makes the application get stuck, but not sure which one. After accepting a memory dump for a web application, we see very few threads working, because the thread is returning to the thread pool due to the use of async / await. The question is, is it possible to list current tasks in a memory dump and where do they work so that I can determine which async operation is causing the application to get stuck? For synchronization blocking calls, this is simple - just list the call stacks of all active threads. But for asynchronous operation, it no longer works. (Adding more traces is a possible approach, but the situation is that we cannot guarantee that we have enough traces for each asynchronous operation in the application and its dependent libraries.)

For example, if an ASP.Net Core application is stuck in some code like this, how can I say this from a memory dump?

public async Task SomeBadMethodInADependentLibrary() { TaskCompletionSource<int> tcs = new TaskCompletionSource<int>(); await tcs.Task; } 

Answer 1

You can, of course, find task objects on the heap and start analyzing them manually using SOS commands, for example. how is the start of a debugging session:

 0:013> !dumpheap -stat -type Task Statistics: MT Count TotalSize Class Name [...] 71e03f28 4 160 System.Threading.Tasks.Task Total 28 objects 0:013> !dumpheap -mt 71e03f28 Address MT Size 022bd900 71e03f28 40 [...] 0:013> !do 022bd900 Name: System.Threading.Tasks.Task MethodTable: 71e03f28 EEClass: 719cd6e0 Size: 40(0x28) bytes File: C:\Windows\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll Fields: MT Field Offset Type VT Attr Value Name 71df1638 40019cd 1c System.Int32 1 instance 3 m_taskId 71defb44 40019ce 4 System.Object 0 instance 022bd8e0 m_action [...] 0:013> !DumpObj 022bd8e0 Name: System.Action MethodTable: 71e0588c EEClass: 719357b8 Size: 32(0x20) bytes File: C:\Windows\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll Fields: MT Field Offset Type VT Attr Value Name 71defb44 40002b5 4 System.Object 0 instance 022bd8e0 _target 71defb44 40002b6 8 System.Object 0 instance 00000000 _methodBase 71df2bdc 40002b7 c System.IntPtr 1 instance 4b00e64 _methodPtr 71df2bdc 40002b8 10 System.IntPtr 1 instance 4e0c30 _methodPtrAux [...] 0:013> !u 4e0c30 Unmanaged code 004e0c30 e833df8372 call clr!PrecodeFixupThunk (72d1eb68) [...] 

And now it starts cumbersome ...

The most convenient way (in WinDbg) from my point of view is to use the !TaskTriage Mex (Github) command:

 0:013> !TaskTriage Normal Mode - not showing successful Tasks Address Target Status Method Exceptions ================================================================================================== 022bd900 | 022bd8e0 | TASK_STATE_DELEGATE_INVOKED | Demo.Program.printMessage() | <none> 022bd974 | 022bd868 | TASK_STATE_DELEGATE_INVOKED | Demo.Program+<>c.<Main>b__0_0() | <none> 022bd9bc | 022bd868 | TASK_STATE_STARTED | Demo.Program+<>c.<Main>b__0_1() | <none> 022bda04 | 022bd868 | TASK_STATE_STARTED | Demo.Program+<>c.<Main>b__0_2() | <none> ================================================================================================== Address Target Status Method Exceptions 

The idea of ​​using WinDbg over Visual Studio is good, since VS2015 and VS2017 will not be able to give the same results from the dump file: