Diagnosing OS X kernel panic. How to translate return addresses

Question

I am debugging a driver that causes a kernel dump on my mac. It shows a long backtrace made up of a series of addresses:

panic(cpu 6 caller 0xffffff8004dc9986): trying to interlock destroyed mutex (0xffffff8049deedb0) Backtrace (CPU 6), Frame : Return Address 0xffffff93b1c8bb50 : 0xffffff8004ce5307 0xffffff93b1c8bbd0 : 0xffffff8004dc9986 0xffffff93b1c8bbe0 : 0xffffff8004d099eb 0xffffff93b1c8bc20 : 0xffffff7f85604899 0xffffff93b1c8bc50 : 0xffffff800519776b 0xffffff93b1c8bc90 : 0xffffff80051f336c 0xffffff93b1c8be00 : 0xffffff8005205fb3 0xffffff93b1c8bef0 : 0xffffff80052028a6 0xffffff93b1c8bf60 : 0xffffff800522afd1 0xffffff93b1c8bfb0 : 0xffffff8004df4b16 

I assume that since the computer was rebooted, address translation is useless, as memory mapping may differ after each iteration of the load.

Is there a way to map the corresponding method to each address in retrospect or to set the appropriate configuration in advance? thanks

Answer 1

Yes, you can definitely symbolize traces of a kernel panic, but to do this retroactively, you need more information from the panic log than just tracing the original stack. As you say, addresses only make sense regarding download addresses.

Apple's official documentation on this topic TN2063 is a bit outdated. The example he gives is Darwin 9, that is, OS X 10.5, and since then the situation has changed a bit with the introduction of ULID Kernel ASLR and kext UUID. I will try to give you a very quick reference.

1. Easy way

If your panic is reproducible, the easiest way is to make the kernel symbolize it for you. Using the kernel boot argument keepsyms=1 means that the kernel will not discard any characters stored in the kernel and kext images, and will look for return pointers in the stack trace in case of panic.

Just add keepsyms=1 to the "Kernel flags" value in /Library/Preferences/SystemConfiguration/com.apple.Boot.plist or to the boot-args NVRAM variable. Reboot, and any subsequent panic will be automatically indicated. You can run C ++ managed characters using the c++filt command-line utility to get the correct C ++ function signatures. For instance,

 $ echo __ZN32IOPCIMessagedInterruptController17registerInterruptEP9IOServiceiPvPFvS2_S2_S2_iES2_ | c++filt IOPCIMessagedInterruptController::registerInterrupt(IOService*, int, void*, void (*)(void*, void*, void*, int), void*) 

2. Manual way

If you have an asymptomatic, mysterious panic that you cannot reproduce, an easy way does not help much.

Immediately after the stack trace, locate the section starting with "Kernel extensions in backtrace:" in the panic log. This list will indicate any kexts participating in the panic, including their download addresses, versions, and UUIDs. Addresses are indicated as a range; the start address is to the left of -> and after @ . The last address is to the right of the arrow. Using this information, you can determine the kext in which each code address is located (the hexadecimal number on the right) specified in the stack trace.

In addition, some of them will not match any kext. Except for some strange circumstances, this will be from the core itself. The boot address of the kernel image (kernel or mach_kernel) is located below where it says "Kernel text base:"

Once you know what the executable image will look like, the atos command will let you symbolize each address.

For example, suppose we have this panic in a panic:

 … 0xffffff8098c1bba0 : 0xffffff7f80c343f2 … 

And we also find that:

  Kernel Extensions in backtrace: com.apple.iokit.IOPCIFamily(2.9)[BDA92C3B-AD86-33E5-A7F1-1603465350A7]@0xffffff7f80c1a000->0xffffff7f80c4dfff 

Note that 0xffffff7f80c343f2 is greater than (or equal to) 0xffffff7f80c1a000 and less than (or equal to) 0xffffff7f80c4dfff, so this code is in IOPCIFamily.

This leads me to the following command (and its output):

 $ atos -o /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Extensions/IOPCIFamily.kext/IOPCIFamily -l 0xffffff7f80c1a000 0xffffff7f80c343f2 IOPCIMessagedInterruptController::registerInterrupt(IOService*, int, void*, void (*)(void*, void*, void*, int), void*) (in IOPCIFamily) (IOPCIMessagedInterruptController.cpp:85) 

-o indicates the executable file. This is usually located in the Contents / MacOS / .kext subdirectory, but some of them Apple kexts are located directly in the .kext directory. For functions in the kernel itself, provide a kernel image, for example. /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel .

The -l argument specifies the download address. That is, the beginning / text of the database.

Finally, just list all the addresses you want to symbolize in this file. In this case, only one, but you can list a few. You can also read them from stdin (if none of them are listed on the command line).

In this case, you can decode your entire trace.

UUID Note

You will notice that each kext in the trace and the kernel itself contains a list of UUIDs. This is convenient for providing the correct version for symbology. This is the UUID from the LC_UUID loader command in the Mach-O binary. You can check the UUID for kext using:

 $ otool -l /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Extensions/IOPCIFamily.kext/IOPCIFamily | grep uuid uuid BDA92C3B-AD86-33E5-A7F1-1603465350A7 

To confirm that kext is used for symbolism, it really matches that in a panic. This is great when you end up with strange versioning issues or have problems with the kext cache.