Linux theme specific topic data

How to get a pointer to a streaming local storage or data with specific threads when analyzing a kernel dump for linux ?

I use pthread_setspecific to store some data in local stoare pthread.

my multithreaded linux program crashed and I want to see what is stored in the current current local thread store.

If I get a pointer to streaming local storage, I can use the key to get the data that is stored.

Is there a command in gdb to get a pointer for streaming local storage ?

+6
source share
2 answers

If you are debugging a live program, you can:

print pthread_getspecific(i) 

If you have access to the pthread_t thread, you can:

 print ((struct pthread*)pth)->specific[i/32][i%32] 

where I am at the index you want, and pth is pthread_t. See Nptl / pthread_getspecific.c in glibc sources.

To do this without calling a function, you need to find struct pthread. On x86-64, it is stored in the fs database, which is set using arch_prctl (ARCH_SET_FS_BASE, ...). I don't know how to access this from gdb, but you can get it with eu-readelf. Run eu-readelf --notes core_file and view the entries for fs.base . This number is the pthread_t value. (To find out which one is there, you can map the pid field in the same entry to the LWP shown in the gdb info threads command.)

Good luck

+4
source

As far as I know, in gdb there is no command to get a pointer to the data stored through pthread_setspecific() . However, there are several options for obtaining a memory address:

  • Examine each thread back trace by checking each frame to see if the result of pthread_getspecific() on the stack.
  • Modify the existing code to register both the thread identifier and the result of pthread_getspecific() .
  • Find the list of stream related data inside the internal stream data, then use the key to find the entry that will contain the address. This approach depends on the implementation of the pthread library; thus, this requires either knowledge of the pthread implementation, or reverse engineering with little patience.

Below is a demo with a simple program on a 32-bit machine:

  • g ++ (GCC) 4.1.2 20080704 (Red Hat 4.1.2-48)
  • libpthread-2.5.so
  • GNU gdb Red Hat Linux (6.5-25.el5rh)

 $cat example.cpp #include <pthread.h> #include <stdio.h> #include <stdlib.h> #include <unistd.h> void* the_thread( void* ); void get_position(); struct position_t { int x; int y; }; namespace { pthread_key_t position_key; enum { NUMBER_OF_THREADS = 2 }; } // unnamed int main(int argc, char **argv) { int result = pthread_key_create( &position_key, NULL ); printf( "pthread_key_create -- key: %u, result: %i\n", position_key, result ); pthread_t threads[NUMBER_OF_THREADS]; for (unsigned int i = 0; i < NUMBER_OF_THREADS; ++i ) { // Allocate a position per threads. position_t* position = new position_t(); // Set position values. position->x = ( 1 + i ) * 11; position->y = ( 1 + i ) * 13; // Create the thread. result = pthread_create( &threads[i], NULL, the_thread, position ); } // Give time for threads to enter their forever loop. sleep( 5 ); // Abort. abort(); return 0; } void* the_thread( void* position ) { int result = pthread_setspecific( position_key, position ); printf( "Thread: 0x%.8x, key: %u, value: 0x%.8x, result: %i\n", pthread_self(), position_key, position, result ); get_position(); return 0; } void get_position() { position_t* position = reinterpret_cast< position_t* >( pthread_getspecific( position_key ) ); printf( "Thread: 0x%.8x, key: %u, position: 0x%.8x, x: %i, y: %i\n", pthread_self(), position_key, position, position->x, position->y ); // Wait forever. while( true ) {}; } $ g++ -g -lpthread example.cpp && gdb -q ./a.out Using host libthread_db library "/lib/libthread_db.so.1". (gdb) r Starting program: /tmp/a.out [Thread debugging using libthread_db enabled] [New Thread -1209043248 (LWP 17390)] pthread_key_create -- key: 0, result: 0 [New Thread -1209046128 (LWP 17393)] Thread: 0xb7ef6b90, key: 0, value: 0x09a35008, result: 0 Thread: 0xb7ef6b90, key: 0, position: 0x09a35008, x: 11, y: 13 [New Thread -1219535984 (LWP 17394)] Thread: 0xb74f5b90, key: 0, value: 0x09a350b0, result: 0 Thread: 0xb74f5b90, key: 0, position: 0x09a350b0, x: 22, y: 26 Program received signal SIGABRT, Aborted. [Switching to Thread -1209043248 (LWP 17390)] 0x00377402 in __kernel_vsyscall () 

Using addresses still on the stack:

 (gdb) info threads 3 Thread -1219535984 (LWP 17394) get_position () at example.cpp:71 2 Thread -1209046128 (LWP 17393) get_position () at example.cpp:71 * 1 Thread -1209043248 (LWP 17390) 0x00377402 in __kernel_vsyscall () (gdb) thread 3 [Switching to thread 3 (Thread -1219535984 (LWP 17394))]#0 get_position () at example.cpp:71 71 while( true ) {}; (gdb) list get_position 57 58 get_position(); 59 return 0; 60 } 61 62 void get_position() 63 { 64 position_t* position = 65 reinterpret_cast< position_t* >( pthread_getspecific( position_key ) ); 66 (gdb) info locals position = (position_t *) 0x9a350b0 (gdb) p position->x $1 = 22 (gdb) p position->y $2 = 26 

Using addresses printed with stdout:

 (gdb) p ((position_t*)(0x09a350b0))->x $3 = 22 (gdb) p ((position_t*)(0x09a350b0))->y $4 = 26 

Find the list of data related to the stream inside the internal data of the streams:

This approach is much simpler if you have the value key and pthread_t .

I will give details about the implementation of pthread, which I use as necessary:

  • pthread struct is the thread descriptor structure used internally by pthread.
  • pthread_create() returns pthread_t , a unsigned int , which contains the address of the associated pthread structure.

First find the pthread structure for the thread.

 (gdb) info threads * 3 Thread -1219535984 (LWP 17394) get_position () at example.cpp:71 2 Thread -1209046128 (LWP 17393) get_position () at example.cpp:71 1 Thread -1209043248 (LWP 17390) 0x00377402 in __kernel_vsyscall () (gdb) thread 1 [Switching to thread 1 (Thread -1209043248 (LWP 17390))]#0 0x00377402 in __kernel_vsyscall () (gdb) bt #0 0x00377402 in __kernel_vsyscall () #1 0x0080ec10 in raise () from /lib/libc.so.6 #2 0x00810521 in abort () from /lib/libc.so.6 #3 0x0804880f in main () at example.cpp:47 (gdb) frame 3 #3 0x0804880f in main () at example.cpp:47 47 abort(); (gdb) info locals result = 0 threads = {3085921168, 3075431312} (gdb) p/x threads[1] $5 = 0xb74f5b90 

Ignoring many fields, the pthread structure pthread as follows:

 struct pthread { ... pid_t tid; // Thread ID (ie this thread descriptor). pid_t pid; // Process ID. ... struct pthread_key_data { uintptr_t seq; void *data; } specific_1stblock[PTHREAD_KEY_2NDLEVEL_SIZE]; struct pthread_key_data *specific[PTHREAD_KEY_1STLEVEL_SIZE]; ... }; 
  • pthread_key_data.seq : contains a sequence number that should be pretty low and match __pthread_keys[key].seq .
  • pthread_key_data.data : contains the value specified in pthread_setspecific()
  • pthread.specific_1stblock is a block that is used to store data that depends on the thread before trying to dynamically allocate more blocks.
  • pthread is a two-level array for thread related data. Index 0 will contain the memory address pthread.specific_1stblock .
  • PTHREAD_KEY_2NDLEVEL_SIZE has a size of 32.

The definition gives a good idea of ​​what to look for in memory:

  • An integer with the value of the memory address pthread ( tid ), followed by an integer with the process identifier ( pid ). This is useful to indicate whether the memory in pthread structure.
  • cancelhandling and flags are flags. Specific values ​​are not important. These fields can be useful because their values ​​are potentially noticeable from other fields, such as those that contain memory addresses or counters.
  • specific_1stblock is an array of size 32. If the pthread structure was initialized to zero, then you need to repeat 0 for 62 ~ words, because the sample code has only one position_key data stream that is two words in size.
  • specific - an array containing memory addresses. If the pthread structure was initialized to zero, then you need to repeat 0 s, but the first value should be the memory address specific_1stblock .

Print out the pthread :

 (gdb) p/x *((int*)threads[1])@150 $6 = {0xb74f5b90, 0x9a350c8, 0xb74f5b90, 0x1, 0x377400, 0x7fb99100, 0xcb40329e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb7ef6bd0, 0x96b118, 0x43f2, 0x43ee, 0xb74f5be0, 0xffffffec, 0x0, 0x0, 0xb74f5470, 0x0, 0x1, 0x9a350b0, 0x0 <repeats 62 times>, 0xb74f5bf8, 0x0 <repeats 31 times>, 0x1000101, 0x0, 0x0, 0x0, 0xc2342345, 0xe0286, 0x0, 0x0, 0x0, 0x0, 0x0, 0x80486ca, 0x9a350b0, 0x0 <repeats 13 times>, 0xb6af5000, 0xa01000} 

By analyzing patterns in memory, some words become good candidates for specific pthread fields:

 0xb74f5b90, 0x9a350c8, 0xb74f5b90 (pthread.tid), 0x1, 0x377400 (pthread.pid) ... 0x1, 0x9a350b0, 0x0 <repeats 62 times> (pthread.specific_1stblock) ... 0xb74f5bf8, 0x0 <repeats 31 times> (pthread.specific) 

You can perform several sensitivity checks, for example, if pthread.specific[0] contains the address pthread.specific_1stblock :

 (gdb) p/x *((int*)0xb74f5bf8)@64 $7 = {0x1, 0x9a350b0, 0x0 <repeats 62 times>} ## matches specific_1stblock 

Now that pthread.specific been identified, get its memory address by counting the word offset from &pthread . In this case, it is 90:

 (gdb) set $specific=(int*)threads[1] + 90 

Calculate the first and second indexes using position_key :

  • The index in the first array is key / PTHREAD_KEY_2NDLEVEL_SIZE .
  • The index into the second array is key % PTHREAD_KEY_2NDLEVEL_SIZE .

     (gdb) set $index1=position_key/32 (gdb) set $index2=position_key%32 

Find pthread_key_data for position_key :

 (gdb) set $level2=(int*)*($specific + $index1) (gdb) p/x *($level2 + (2*$index2))@2 $8 = {0x1, 0x9a350b0} 

Thus:

pthread_key_data.seq = 1
pthread_key_data.data = 0x9a350b0

The first word is seq , which must match pthread_key_struct[position_key].seq . Due to raw memory __pthread_keys , __pthread_keys will be added to int* , and pointer arithmetic must occur to calculate the size of pthread_key_struct :

 (gdb) p *(&((int*)&__pthread_keys)[2*position_key])@2 $9 = {1, 0} 

Thus:

pthread_key_struct[position_key].seq = 1
pthread_key_struct[position_key].destr = NULL

The seq number matches, so everything looks good. pthread_key_data.data contains the value to be returned from pthread_getspecific( position_key ) .

 (gdb) set $position=(position_t*)0x9a350b0 (gdb) p $position->x $10 = 22 (gdb) p $position->y $11 = 26 

Technically, you can still find thread-specific data without knowing the key and pthread_t values:

  • If a destructor function was provided for pthread_key_create() , then its memory address may be inside the __pthread_keys array. Examine the memory and calculate the offset and divide by sizeof pthread_key_struct . This should result in an index, which is also the key:

     void* destr_fn( void* ); pthread_key_create( key, destr_fn ) __pthread_keys[key].destr == destr_fn 
  • If pthread_t not known, it may exist in the thread stack register. This may require examining various memory addresses in an attempt to find a section in memory containing the pthread structure.

     (gdb) info thread 3 Thread -1219535984 (LWP 17394) get_position () at example.cpp:71 2 Thread -1209046128 (LWP 17393) get_position () at example.cpp:71 * 1 Thread -1209043248 (LWP 17390) 0x00377402 in __kernel_vsyscall () (gdb) thread 3 [Switching to thread 3 (Thread -1219535984 (LWP 17394))]#0 g get_position () at example.cpp:71 71 while( true ) {}; (gdb) bt #0 get_position () at example.cpp:71 #1 0x0804871d in the_thread (position=0x9a350b0) at example.cpp:58 #2 0x0095c43b in start_thread () from /lib/libpthread.so.0 #3 0x008b3fde in clone () from /lib/libc.so.6 (gdb) frame 2 #2 0x0095c43b in start_thread () from /lib/libpthread.so.0 (gdb) info register eax 0x3f 63 ecx 0xb74f52ac -1219538260 edx 0x0 0 ebx 0x96aff4 9875444 esp 0xb74f53c0 0xb74f53c0 ebp 0xb74f54a8 0xb74f54a8 esi 0x0 0 edi 0xb74f5b90 -1219535984 eip 0x95c43b 0x95c43b <start_thread+203> eflags 0x200286 [ PF SF IF ID ] cs 0x73 115 ss 0x7b 123 ds 0x7b 123 es 0x7b 123 fs 0x0 0 gs 0x33 51 

    In this case, the edi register contains the address of the pthread structure.

Links: descr.h , pthread_key_create.c , pthread_setspecific.c , pthreadP.h , internaltypes.h

+1
source

Source: https://habr.com/ru/post/917130/


All Articles