Is it possible to use systemtap 1.7 / 2.1 on Ubuntu 12.04 with 3.5.0 / 3.8.0 kernels?

Question

My goal is to be able to use user space probes on Ubuntu 12.04 / exact host. This is possible starting with kernel 3.5.0, so I installed the following packages:

linux-headers-3.5.0-34 linux-headers-3.5.0-34-generic linux-image-3.5.0-34-generic linux-image-3.5.0-34-generic-dbgsym linux-image-generic-lts-quantal linux-source linux-source-3.5.0 

and systemtap v1.7 from 12.10 / quantal.

But the only thing I see after running any stap script:

 ERROR: module version mismatch (#55-Ubuntu SMP Thu Jun 6 20:18:19 UTC 2013 vs #55~precise1-Ubuntu SMP Fri Jun 7 16:25:50 UTC 2013), release 3.5.0-34-generic ERROR: Build-id mismatch [man error::buildid]: "/usr/lib/debug/boot/vmlinux-3.5.0-34-generic" byte 0 (0x84 vs 0xc2) address 0xffffffff8168e60c rc 0 

uname -a:

 Linux nginx-dev01g 3.5.0-34-generic #55~precise1-Ubuntu SMP Fri Jun 7 16:25:50 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux 

systemtap v2.1 dated 13.04 / raring shows exactly the same error.

systemtap v1.6 (the default version is 12.04) does not work at compile time:

 In file included from /usr/share/systemtap/runtime/transport/transport.c:53:0, from /usr/share/systemtap/runtime/print.c:18, from /usr/share/systemtap/runtime/runtime.h:128, from /tmp/stapYh6Wqv/stap_c22f4bc3b03cd6f9fed549d2973f11fd_361755.c:59: /usr/share/systemtap/runtime/transport/relay_v2.c:241:2: error: initialization from incompatible pointer type [-Werror] /usr/share/systemtap/runtime/transport/relay_v2.c:241:2: error: (near initialization for '__stp_relay_callbacks.create_buf_file') [-Werror] In file included from /usr/share/systemtap/runtime/print.c:18:0, from /usr/share/systemtap/runtime/runtime.h:128, from /tmp/stapYh6Wqv/stap_c22f4bc3b03cd6f9fed549d2973f11fd_361755.c:59: /usr/share/systemtap/runtime/transport/transport.c: In function '_stp_get_root_dir': /usr/share/systemtap/runtime/transport/transport.c:386:8: error: 'struct hlist_head' has no member named 'next' In file included from /tmp/stapYh6Wqv/stap_c22f4bc3b03cd6f9fed549d2973f11fd_361755.c:61:0: /usr/share/systemtap/runtime/stat.c: In function '_stp_stat_get': /usr/share/systemtap/runtime/stat.c:213:2: error: 'cpu_possible_map' undeclared (first use in this function) /usr/share/systemtap/runtime/stat.c:213:2: note: each undeclared identifier is reported only once for each function it appears in /usr/share/systemtap/runtime/stat.c: In function '_stp_stat_clear': /usr/share/systemtap/runtime/stat.c:248:2: error: 'cpu_possible_map' undeclared (first use in this function) In file included from /tmp/stapYh6Wqv/stap_c22f4bc3b03cd6f9fed549d2973f11fd_361755.c:13594:0: /usr/share/systemtap/runtime/pmap-gen.c: In function '_stp_pmap_new_sx': /usr/share/systemtap/runtime/pmap-gen.c:717:3: error: 'cpu_possible_map' undeclared (first use in this function) /usr/share/systemtap/runtime/pmap-gen.c: In function '_stp_pmap_get_sx': /usr/share/systemtap/runtime/pmap-gen.c:943:2: error: 'cpu_possible_map' undeclared (first use in this function) In file included from /usr/share/systemtap/runtime/map.c:20:0, from /tmp/stapYh6Wqv/stap_c22f4bc3b03cd6f9fed549d2973f11fd_361755.c:13609: /usr/share/systemtap/runtime/map-stat.c: In function '_stp_pmap_new_hstat_linear': /usr/share/systemtap/runtime/map-stat.c:71:3: error: 'cpu_possible_map' undeclared (first use in this function) /usr/share/systemtap/runtime/map-stat.c: In function '_stp_pmap_new_hstat_log': /usr/share/systemtap/runtime/map-stat.c:98:3: error: 'cpu_possible_map' undeclared (first use in this function) In file included from /tmp/stapYh6Wqv/stap_c22f4bc3b03cd6f9fed549d2973f11fd_361755.c:13609:0: /usr/share/systemtap/runtime/map.c: In function '_stp_pmap_new': /usr/share/systemtap/runtime/map.c:242:2: error: 'cpu_possible_map' undeclared (first use in this function) /usr/share/systemtap/runtime/map.c: In function '_stp_pmap_clear': /usr/share/systemtap/runtime/map.c:350:2: error: 'cpu_possible_map' undeclared (first use in this function) /usr/share/systemtap/runtime/map.c: In function '_stp_pmap_del': /usr/share/systemtap/runtime/map.c:402:2: error: 'cpu_possible_map' undeclared (first use in this function) /usr/share/systemtap/runtime/map.c: In function '_stp_pmap_agg': /usr/share/systemtap/runtime/map.c:732:2: error: 'cpu_possible_map' undeclared (first use in this function) /usr/share/systemtap/runtime/map.c: In function '_stp_pmap_size': /usr/share/systemtap/runtime/map.c:908:2: error: 'cpu_possible_map' undeclared (first use in this function) cc1: all warnings being treated as errors 

Unfortunately, updating to 12.10 / 13.04 is not an option.

So, is there a chance to get a working system table with kernels 3.5.0 or 3.8.0 on Ubuntu 12.04 / exact?

Answer 1

You may need to update your version of systemtap, since a very old one (for example, 1.7 - since the beginning of 2012) often does not work with a very new kernel (but vice versa is excellent).

OTOH, the original build-id error is the real version mismatch issue in the ubuntu -dbgsym repositories. You must make sure your kernel version works exactly the same as dbgsym. With Ubuntu, this can be tricky because "uname -r" and dpkg names do not contain a unique identifier for the assembly. See the difference between:

 #55-Ubuntu SMP Thu Jun 6 20:18:19 UTC 2013 

and

 #55~precise1-Ubuntu SMP Fri Jun 7 16:25:50 UTC 2013 

Why do you need kernel debugging symbols for user space probes? Please provide your script stamp and stap-report output. Try copying the script as well.

Another complicating factor is the possible absence of CONFIG_UPROBES in the Ubuntu kernel. See also http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=691167

Answer 2

I compiled a version of systemstap-2.3. It runs on Ubuntu 12.04 with linux-image-3.5.0-40-generic-dbgsym.

Install the kernel with debugging information. Where to get debug symbols for kernel X?

Compile system file 2.3

 sudo apt-get build-dep systemtap git clone git://sourceware.org/git/systemtap.git cd systemtap git checkout release-2.3 ./configure make all # you can ignore the xmlto error # enjoy it sudo ./stap testsuite/systemtap.examples/network/tcpdumplike.stp