Epoll_wait () gets a socket closed twice (read () / recv () returns 0)

Question

We have an application that uses epoll to listen and process http connections. Sometimes epoll_wait () gets a private event on fd twice in a row. Value: epoll_wait () returns a fd connection to which read () / recv () returns 0. This is a problem, since I have malloc: ed pointer stored in epoll_event struct (struct epoll_event.data.ptr) and which is freed, when fd (socket) is defined as closed for the first time. The second time he falls.

This problem is very rare in real use (with the exception of one site, which actually has about 500-1000 users per server). I can replicate the problem using http siege s> 1000 concurrent connections per second. In this case, the segfaults application (due to an invalid pointer) happens in a very random way, sometimes after a few seconds, usually after a few tens of minutes. I was able to replicate the problem with fewer connections per second, but for this I have to run the application for a long time, many days, even weeks.

The whole new connection accept () fd: s is set as non-blocking and is added to epoll as a one-time, front-trigger and wait for read (). So when the server load is high, epoll thinks that my application has not received the closure and the order of the new?

epoll_wait () starts its own thread in it and queues fd events that need to be processed elsewhere. I noticed that there were several closures coming in with simple code that checks to see if an event occurs twice in a line from epoll to the same fd. This happened and the events that both close (recv (.., MSG_PEEK) told me that :)).

epoll fd is created:

epoll_create (1024);

epoll_wait () starts as follows:

epoll_wait (epoll_fd, events, 256, 300);

new fd is set as non-blocking after accept ():

int flags = fcntl (fd, F_GETFL, 0);
err = fcntl (fd, F_SETFL, flags | O_NONBLOCK);

added new fd to epoll (client is a struct moldoc: ed struct pointer):

static struct epoll_event ev;
ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET;
ev.data.ptr = client;
err = epoll_ctl (epoll_fd, EPOLL_CTL_ADD, client-> fd, & ev);

fd, (, EPOLLONESHOT). - io, , . . Btw. shutdown (fd, SHUT_RDWR) , , epoll, fd - HTTP- .. ( , , ).

Answer 1

read() 0, , . ​​ EPOLLIN ? , , EPOLLIN. EPOLLRDHUP, , (), 0. EPOLLIN , .

  if (flag & EPOLLRDHUP) {
     /* Connection was closed. */
     deleteConnectionData(...);
     close(fd); /* Will unregister yourself from epoll. */
     return;
  }

  if (flag & EPOLLIN) {
    readData(...);
  }

  if (flag & EPOLLOUT) {
    writeData(...);
  }

, , , EPOLLRDHUP , , , deleteConnectionData() . EPOLLIN , . EPOLLIN , . EPOLLOUT, EPOLLRDHUP!

Answer 2

epoll_wait() fd, .... , , , epoll , ?

, EPOLLONESHOT ( , ), , epoll , , - .

, , epoll_event.data.ptr, , epoll , .

, valgrind , - .

Answer 3

: epoll(7):

Q6

, epoll?

o ...

.

Answer 4

EPOLLONESHOT . , , . EPOLLONESHOT fd epoll . epoll struct . .

Answer 5

Register 0x2000 signal for remote connection of a remote host ex ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET | 0x2000 and make sure (flag and 0x2000) to connect the remote host