WinDbg takes a lot of time to load characters; looks for each directory in the UNC large network character store

Question

For several days I tried to speed up the loading of characters when debugging crash dumps using WinDbg , and I can not get past a particular problem.

The problem is that when the symbols for the module in the dump are absent in any available symbol storage or in the location of the symbol server (for example, these are third-party modules without available symbols), WinDbg will spend literally hours searching for them.

I set my character path correctly to correctly set the search order and cache directories:

.sympath cache*C:\SymbolCache1;\\our.corp\SymbolStore;SRV*C:\SymbolCache2*http://msdl.microsoft.com/download/symbols 

Starting with !sym noisy and .reload /f I see:

 SYMSRV: Notifies the client application that a proxy has been detected. SYMSRV: Connecting to the Server: http://msdl.microsoft.com/download/symbols. SYMSRV: Successfully connected to the Server. SYMSRV: Sending the information request to the server. SYMSRV: Successfully sent the information request to the server. SYMSRV: Waiting for the server to respond to a request. SYMSRV: Successfully received a response from the server. SYMSRV: Closing the connection to the Server. SYMSRV: Successfully closed the connection to the Server. SYMSRV: c:\SymbolCache1\Some3rdParty.dll\0060D200cd1000\Some3rdParty.dll not found SYMSRV: c:\SymbolCache2\Some3rdParty.dll\0060D200cd1000\Some3rdParty.dll not found SYMSRV: http://msdl.microsoft.com/download/symbols/Some3rdParty.dll/0060D200cd1000/Some3rdParty.dll not found <---- !!!! hanging here with *BUSY* showing in WinDbg 

Running Process Monitor in the place where it hangs, I see that WinDbg is looking for what it looks like in every directory in our giant network store of characters (\ our.corp \ SymbolStore), which looks for characters, even in directories for modules, which are clearly unrelated.

What is strange is that in WinDbg you can see that it has extracted the timestamp of the module (0060D200cd1000) and uses it to search in the expected location in local directories and the MS symbol server. I can’t understand why it is doing a full check of our (massive) character store on the network. Maybe there is something unique about how it relates to UNC paths?

This search may take 15 minutes or more per character, and if the dump has a lot of missing characters, it may cause !analyze -v take hours (if you use the WinDbg integration in Visual Studio, this will cause a crash as soon as you are loading a crash dump, because for some reason the integration is trying to load all the characters immediately, despite the .symopt settings).

This problem is also easily reproducible if you are trying to load characters for a non-existent module name, for example. .reload /f bogus.dll .

Here are my WinDbg.symopt settings:

 0:000> .symopt Symbol options are 0x30337: 0x00000001 - SYMOPT_CASE_INSENSITIVE 0x00000002 - SYMOPT_UNDNAME 0x00000004 - SYMOPT_DEFERRED_LOADS 0x00000010 - SYMOPT_LOAD_LINES 0x00000020 - SYMOPT_OMAP_FIND_NEAREST 0x00000100 - SYMOPT_NO_UNQUALIFIED_LOADS 0x00000200 - SYMOPT_FAIL_CRITICAL_ERRORS 0x00010000 - SYMOPT_AUTO_PUBLICS 0x00020000 - SYMOPT_NO_IMAGE_SEARCH 

I kept thinking that there should be some kind of flag to control this, but I can’t find it.

A few things:

• This is not a problem with network speed or the lack of a local character cache. The problem occurs only with characters that cannot be found, and only with the UNC character store (for example, not with the Microsoft character server).
• I already tried SYMOPT_NO_PUBLICS instead of SYMOPT_AUTO_PUBLICS
• I checked that my symbol path is what I expect using the sympath command. I also tried using the _NT_SYMBOL_PATH environment _NT_SYMBOL_PATH .
• I know that I can exclude certain characters through the configuration file, but this is not a feasible solution, because sometimes the missing character name is not known in advance
• I saw that someone else on the Internet had the same problem and mentioned this on the Microsoft forum in a post titled “ Poor WinDbg 6.11.1.404 performance when it points to a large character cache ”, but received no help.

Answer 1

Must not be:

 .sympath cache*C:\SymbolCache1;SRV*\\our.corp\SymbolStore;SRV*C:\SymbolCache2*http://msdl.microsoft.com/download/symbols 

That is, listing \\our.corp\SymbolStore without SRV* , you say dbghelp to look at the characters in this unstructured directory. If you use the SRV * syntax, then you tell dbghelp to get symsrv to look in this directory for a very specific and structured way.

symsrv.dll can efficiently search the Microsoft Symbol Server, and it can efficiently search your data if you tell it to do it using SRV* .

Answer 2

Finally, I found an unsatisfactory but perfectly suitable solution: configure SymProxy .

This will allow me to remove the UNC resource from my symbol path and replace it with a link to SymProxy as my http symbol server.

 .sympath SRV*C:\SymbolCache*http://somemachine.our.corp/Symbols 

The proxy server itself still searches the UNC shared folder, but WinDbg can no longer search the network directory - instead, it must pass information about the character that it wants to use for SymProxy, and SymProxy looks at exactly one location on the UNC instead to perform an exhaustive search.

This does not explain why WinDbg searches for the entire UNC resource, but it fixes the problem of hanging WinDbg while searching for characters. Finally, the loading of characters is repeated again!

Another advantage of installing SymProxy is that you can configure it to migrate from multiple character locations. For example, you can connect it to both the local repository of the organization’s symbols and the Microsoft symbol server. Your developers can then set their symbol path to link only to SymProxy, and not to multiple symbol locations.