Segmentation error when connecting to open ()

Question

I am trying to create a binding to the open() system function. I did this in the following lines.

I created a wrapper library with the following:

 extern int mocked_open(const char* fn, int flags, va_list args); int open(const char* fn, int flags, ...) { int r = -1; va_list args; va_start(args, flags); r = mocked_open(fn, flags, args); va_end(args); return r; } 

I will compile this in libwrapper.so, which I load with LD_PRELOAD.

The implementation of mocked_open() as follows (I use the CPPUtest framework):

 int mocked_open(const char* fn, int flags, va_list args) { if (strncmp(fn, test_device_id, 11) == 0) { return mock().actualCall("open").returnValue().getIntValue(); } else { int r = -1; int (*my_open)(const char*, int, ...); void* fptr = dlsym(RTLD_NEXT, "open"); memcpy(&my_open, &fptr, sizeof(my_open)); if (flags & O_CREAT) { r = my_open(fn, flags, va_arg(args, mode_t)); } else { r = my_open(fn, flags); } return r; } } 

test_device_id is a simple string ("test_device"), which I hope is not used elsewhere.

During tests, the executable fails with a segmentation error. I followed this up to the GCC profiling function, which wants to open / create a bunch of .gcda files and calls open() to do this.

After some debugging with strace (as suggested below), I found that the line r = my_open(fn, flags, va_arg(args, mode_t)); really is the culprit. It is called recursively, or so it seems: I see many calls to this line without returning a function. Then segfault. The file to be opened is the corresponding .gcda file (for profiling). In fact, segfault only happens when profiling is enabled.

Answer 1

When compiling with gcov profiling enabled, the compiler adds extra code to your functions to keep track of which code has been executed. In the rough pseudo-code, this inserted code will be (among other things):

 if (!output_file_has_been_opened) { fd = open(output_filename, ...); check_ok(fd); output_file_has_been_opened = TRUE; track_coverage(); } 

... so if the output file has not yet been successfully opened (as at the beginning of your program), it will try to open it. Unfortunately, in this case, it will call your mocked open() function, which has the same code inserted; since the file was still not successfully opened, and since the gcov code does not know that something unusual is happening, it will try to call the open() call again - and this is what causes the recursion (and possibly segfault, once the stack is exhausted )

Answer 2

try it

 typedef int (*OpenFunction)(const char* fn, int flags, ...); 

then

 OpenFunction function; void **pointer; pointer = (void **)&function; *pointer = dlsym(RTLD_NEXT, "open"); 

this is a complete working example

 #include <stdio.h> #include <string.h> #include <stdlib.h> #include <dlfcn.h> #include <unistd.h> #include <fcntl.h> #include <errno.h> typedef int (*OpenFunction)(const char* fn, int flags, ...); int main(int argc, char **argv) { OpenFunction function; void *dl; int fd; void **pointer; if (argc < 2) return -1; pointer = (void **)&function; *pointer = dlsym(RTLD_NEXT, "open"); fd = function(argv[1], O_RDONLY); if (fd != -1) { printf("file opened succesfully\n"); close(fd); } else { printf("%s: cannot open the file\n", strerror(errno)); } return 0; } 

Answer 3

You are misplacing va_list. Try the following hope, it will help

  r = my_open(fn, flags, args); 

For more information http://c-faq.com/varargs/handoff.html