Linux async IO with libaio performance issue

Question

I am trying Linux libaio to optimize I / O performance in a server application. I believe that I did everything necessary (using O_DIRECT, align the buffer with the memory page ...). I expect the io_submit call to return immediately, but a simple test, showing that it actually takes about 80 microseconds, returns to my main i7 laptop. Am I expecting too much or is there something wrong with my test program? (compiled with g ++ --std = C ++ 0x -laio)

#include <unistd.h> #include <fcntl.h> #include <libaio.h> #include <errno.h> #include <cstdlib> #include <cstdio> #include <iostream> #include <chrono> // Open the file for write, return the file descriptor int open_write(char const* file) { int fd = open(file, O_DIRECT|O_CREAT|O_WRONLY, S_IRWXU|S_IRWXG|S_IROTH); if (fd < 0) { perror("open_write"); exit(1); } } // Make a buffer of _size_ byte, fill with 'a', return the buffer, it should be aligned to memory page void* make_write_buffer(size_t size) { void* buf = 0; int ret = posix_memalign((void**)&buf, sysconf(_SC_PAGESIZE), size); if (ret < 0 || buf == 0) { perror("make_write_buffer"); exit(1); } memset(buf, 'a', size); return buf; } int main (int argc, char *argv[]) { static const size_t SIZE = 16 * 1024; // Prepare file and buffer to write int write_fd = open_write("test.dat"); void* buf = make_write_buffer(SIZE); // Prepare aio io_context_t ctx; memset(&ctx, 0, sizeof(ctx)); const int maxEvents = 32; io_setup(maxEvents, &ctx); iocb *iocbpp = new iocb; io_prep_pwrite(iocbpp, write_fd, buf, SIZE, 0); using namespace std::chrono; // Submit aio task auto start = monotonic_clock::now(); int status = io_submit(ctx, 1, &iocbpp); if (status < 0) { errno = -status; perror("io_submit"); exit(1); } auto dur = duration_cast<microseconds>(monotonic_clock::now() - start); std::cout << "io_submit takes: " << dur.count() << " microseconds." << std::endl; io_event events[10]; int n = io_getevents(ctx, 1, 10, events, NULL); close(write_fd); io_destroy(ctx); delete iocbpp; free(buf); return 0; } 

Answer 1

In short: io_submit blocked, and nothing can be done about it except to fix the kernel.

Here is the topic "io_submit blocking" from 2011-09-23 on the linux-aio mailing list.

As indicated in this thread, you can try increasing /sys/block/xxx/queue/nr_requests

Answer 2

How long does io_getevents ? If most of the time is spent in io_submit, and not in io_getevents , then the ion can actually be executed during io_submit . (In my case, I suspect an ext4s error ...)

Another thing you can try is to bind your process to the kernel using taskset ?

Btw you can also get these timings with strace -T .

Edit: I was mistaken, suspecting ext4, it turns out there is no O_DIRECT flag causing synchronous behavior.

Answer 3

As @Arvid pointed out in another answer , io_submit() can block. There is an incomplete list of reasons why io_submit() can block the io_submit() asynchronous I / O delay on Ubuntu Linux , and here I would suggest that recordings with the file extension are performed on the ext4 file system, as a result of which io_submit() works synchronously during recording file system metadata ...