All geek questions in one place

How can I find out the source of this backcrace glibc created with clone ()?

This backtracking comes from a deadlock situation in a multi-threaded application. Other stub threads are blocked inside the malloc () call and appear to wait in this thread.

I don’t understand what this thread creates, since it blocks any functions in my application until it is called:

Thread 6 (Thread 0x7ff69d43a700 (LWP 14191)):
#0  0x00007ff6a2932eec in __lll_lock_wait_private () from /usr/lib64/libc.so.6
#1  0x00007ff6a299460d in _L_lock_27 () from /usr/lib64/libc.so.6
#2  0x00007ff6a29945bd in arena_thread_freeres () from /usr/lib64/libc.so.6
#3  0x00007ff6a2994662 in __libc_thread_freeres () from /usr/lib64/libc.so.6
#4  0x00007ff6a3875e38 in start_thread () from /usr/lib64/libpthread.so.0
#5  0x00007ff6a292534d in clone () from /usr/lib64/libc.so.6

clone () is used to implement fork (), pthread_create (), and possibly other functions. See here and here .

How can I find out if it will work on this track fork(), pthread_create(), signal processor, or something else? Do I just need to dig through glibc code, or can I use gdb or some other tool? Why is this branch needed for internal glibc locking? This would be useful in determining the cause of the impasse.

Additional information and research:

malloc() , () (. this this, malloc() . , , malloc() . , , , . (, ?)

, , ( ) . .

, malloc ?

, :

glibc malloc

Malloc .

?

glibc fork/malloc , ​​ glibc-2.17-162.el7. , - glibc.

( , , . , - (Slurm), .)

EDIT: . 6 - , . Thread 1 pthread_join(). 2-5 malloc(). 7 ( 2-5). , .

Thread 7 (Thread 0x7ff69e672700 (LWP 12650)):
#0  0x00007ff6a291aa3d in poll () from /usr/lib64/libc.so.6
#1  0x00007ff6a3c09064 in _poll_internal (shutdown_time=<optimized out>, nfds=2,
    pfds=0x7ff6980009f0) at ../../../../slurm/src/common/eio.c:364
#2  eio_handle_mainloop (eio=0xf1a970) at ../../../../slurm/src/common/eio.c:328
#3  0x000000000041ce78 in _msg_thr_internal (job_arg=0xf07760)
    at ../../../../../slurm/src/slurmd/slurmstepd/req.c:245
#4  0x00007ff6a3875e25 in start_thread () from /usr/lib64/libpthread.so.0
#5  0x00007ff6a292534d in clone () from /usr/lib64/libc.so.6

Thread 6 (Thread 0x7ff69d43a700 (LWP 14191)):
#0  0x00007ff6a2932eec in __lll_lock_wait_private () from /usr/lib64/libc.so.6
#1  0x00007ff6a299460d in _L_lock_27 () from /usr/lib64/libc.so.6
#2  0x00007ff6a29945bd in arena_thread_freeres () from /usr/lib64/libc.so.6
#3  0x00007ff6a2994662 in __libc_thread_freeres () from /usr/lib64/libc.so.6
#4  0x00007ff6a3875e38 in start_thread () from /usr/lib64/libpthread.so.0
#5  0x00007ff6a292534d in clone () from /usr/lib64/libc.so.6

Thread 5 (Thread 0x7ff69e773700 (LWP 22471)):
#0  0x00007ff6a2932eec in __lll_lock_wait_private () from /usr/lib64/libc.so.6
#1  0x00007ff6a28af7d8 in _L_lock_1579 () from /usr/lib64/libc.so.6
#2  0x00007ff6a28a7ca0 in arena_get2.isra.3 () from /usr/lib64/libc.so.6
#3  0x00007ff6a28ad0fe in malloc () from /usr/lib64/libc.so.6
#4  0x00007ff6a3c02e60 in slurm_xmalloc (size=size@entry=24, clear=clear@entry=false,
    file=file@entry=0x7ff6a3c1f1f0 "../../../../slurm/src/common/pack.c",
    line=line@entry=152, func=func@entry=0x7ff6a3c1f4a6 <__func__.7843> "init_buf")
    at ../../../../slurm/src/common/xmalloc.c:86
#5  0x00007ff6a3b2e5b7 in init_buf (size=16384)
    at ../../../../slurm/src/common/pack.c:152
#6  0x000000000041caab in _handle_accept (arg=0x0)
    at ../../../../../slurm/src/slurmd/slurmstepd/req.c:384
#7  0x00007ff6a3875e25 in start_thread () from /usr/lib64/libpthread.so.0
#8  0x00007ff6a292534d in clone () from /usr/lib64/libc.so.6

Thread 4 (Thread 0x7ff6a4086700 (LWP 5633)):
#0  0x00007ff6a2932eec in __lll_lock_wait_private () from /usr/lib64/libc.so.6
#1  0x00007ff6a28af7d8 in _L_lock_1579 () from /usr/lib64/libc.so.6
#2  0x00007ff6a28a7ca0 in arena_get2.isra.3 () from /usr/lib64/libc.so.6
#3  0x00007ff6a28ad0fe in malloc () from /usr/lib64/libc.so.6
#4  0x00007ff6a3c02e60 in slurm_xmalloc (size=size@entry=24, clear=clear@entry=false,
    file=file@entry=0x7ff6a3c1f1f0 "../../../../slurm/src/common/pack.c",
    line=line@entry=152, func=func@entry=0x7ff6a3c1f4a6 <__func__.7843> "init_buf")
    at ../../../../slurm/src/common/xmalloc.c:86
#5  0x00007ff6a3b2e5b7 in init_buf (size=16384)
    at ../../../../slurm/src/common/pack.c:152
#6  0x000000000041caab in _handle_accept (arg=0x0)
    at ../../../../../slurm/src/slurmd/slurmstepd/req.c:384
#7  0x00007ff6a3875e25 in start_thread () from /usr/lib64/libpthread.so.0
#8  0x00007ff6a292534d in clone () from /usr/lib64/libc.so.6

Thread 3 (Thread 0x7ff69d53b700 (LWP 12963)):
#0  0x00007ff6a2932eec in __lll_lock_wait_private () from /usr/lib64/libc.so.6
#1  0x00007ff6a28af7d8 in _L_lock_1579 () from /usr/lib64/libc.so.6
#2  0x00007ff6a28a7ca0 in arena_get2.isra.3 () from /usr/lib64/libc.so.6
#3  0x00007ff6a28ad0fe in malloc () from /usr/lib64/libc.so.6
#4  0x00007ff6a3c02e60 in slurm_xmalloc (size=size@entry=24, clear=clear@entry=false,
    file=file@entry=0x7ff6a3c1f1f0 "../../../../slurm/src/common/pack.c",
    line=line@entry=152, func=func@entry=0x7ff6a3c1f4a6 <__func__.7843> "init_buf")
    at ../../../../slurm/src/common/xmalloc.c:86
#5  0x00007ff6a3b2e5b7 in init_buf (size=16384)
    at ../../../../slurm/src/common/pack.c:152
#6  0x000000000041caab in _handle_accept (arg=0x0)
    at ../../../../../slurm/src/slurmd/slurmstepd/req.c:384
#7  0x00007ff6a3875e25 in start_thread () from /usr/lib64/libpthread.so.0
#8  0x00007ff6a292534d in clone () from /usr/lib64/libc.so.6

Thread 2 (Thread 0x7ff69f182700 (LWP 19734)):
#0  0x00007ff6a2932eec in __lll_lock_wait_private () from /usr/lib64/libc.so.6
#1  0x00007ff6a28af7d8 in _L_lock_1579 () from /usr/lib64/libc.so.6
#2  0x00007ff6a28a7ca0 in arena_get2.isra.3 () from /usr/lib64/libc.so.6
#3  0x00007ff6a28ad0fe in malloc () from /usr/lib64/libc.so.6
#4  0x00007ff6a3c02e60 in slurm_xmalloc (size=size@entry=24, clear=clear@entry=false,
    file=file@entry=0x7ff6a3c1f1f0 "../../../../slurm/src/common/pack.c",
    line=line@entry=152, func=func@entry=0x7ff6a3c1f4a6 <__func__.7843> "init_buf")
    at ../../../../slurm/src/common/xmalloc.c:86
#5  0x00007ff6a3b2e5b7 in init_buf (size=16384)
    at ../../../../slurm/src/common/pack.c:152
#6  0x000000000041caab in _handle_accept (arg=0x0)
    at ../../../../../slurm/src/slurmd/slurmstepd/req.c:384
#7  0x00007ff6a3875e25 in start_thread () from /usr/lib64/libpthread.so.0
#8  0x00007ff6a292534d in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7ff6a4088880 (LWP 12616)):
#0  0x00007ff6a3876f57 in pthread_join () from /usr/lib64/libpthread.so.0
#1  0x000000000041084a in _wait_for_io (job=0xf07760)
    at ../../../../../slurm/src/slurmd/slurmstepd/mgr.c:2219
#2  job_manager (job=job@entry=0xf07760)
    at ../../../../../slurm/src/slurmd/slurmstepd/mgr.c:1397
#3  0x000000000040ca07 in main (argc=1, argv=0x7fffacab93d8)
    at ../../../../../slurm/src/slurmd/slurmstepd/slurmstepd.c:172
+4
1

start_thread() , pthread_create().

__libc_thread_freeres() - , glibc , . , , , .

arena_thread_freeres() . malloc, . , ( list_lock arena.c).

, (Thread 6).

- pthread_atfork() , fork() . , pthread_atfork() , .

pthread_atfork()? , .

+3

Source: https://habr.com/ru/post/1695894/

More articles:


All Articles