arthrop0d

Brief articles on varied topics.

Linux epoll Readiness Notification

linuxoperating systems

epoll is Linux's kernel interface for waiting until one of many file descriptors—such as a socket or pipe—is ready for I/O. It lets an event loop ask the kernel for the descriptors that need attention instead of repeatedly checking every descriptor itself.

The problem epoll solves

A server may have thousands of client sockets but only a few doing useful work at any moment. Without a readiness mechanism, an event loop would need to try each socket in turn, or repeatedly ask whether each one could be read or written. Most checks would find nothing, and a blocking read on the wrong socket could stall the entire server.

Older interfaces such as select and poll improved this by waiting on a set of descriptors. But the application still supplied the full set on each wait, and the kernel generally examined the set to find changes. epoll keeps the monitored set in the kernel and maintains information about descriptors that have become ready. The application retrieves only the ready entries.

Here, ready means an I/O operation can proceed without blocking. For a socket, EPOLLIN commonly means that a read or recv can return data, or that the peer has closed the connection. EPOLLOUT means that writing can proceed immediately, usually because space is available in the socket's send buffer.

Two kernel-managed lists

An epoll instance has two useful conceptual parts:

The application creates an epoll instance, adds or modifies descriptors with epoll_ctl, and waits with epoll_wait:

int epfd = epoll_create1(EPOLL_CLOEXEC);

struct epoll_event interest = {
    .events = EPOLLIN,
    .data.fd = client_fd,
};
epoll_ctl(epfd, EPOLL_CTL_ADD, client_fd, &interest);

struct epoll_event events[64];
int n = epoll_wait(epfd, events, 64, -1);

When activity changes a descriptor's state, the kernel can place the corresponding entry on the ready list. epoll_wait copies ready events into the application's array and returns. It does not read the socket, accept a connection, or write any data. The event loop must perform that work afterward.

An entry is not a queue of individual bytes or packets. It represents a readiness condition. If ten messages arrive on one socket, epoll does not promise ten separate notifications; it reports that the socket is readable, and the application decides how much to consume.

Level-triggered behavior

By default, epoll is level-triggered (LT). In this mode, epoll continues to report a descriptor while the requested condition remains true.

Suppose a socket has 8 KB waiting to be read, and the handler reads only 1 KB. The socket is still readable, so a later epoll_wait can report it again. This is straightforward and works well with handlers that perform one bounded I/O operation per notification.

The trade-off is that failing to consume the condition causes repeated wakeups. A handler that always reads only a little may repeatedly return to the event loop while data remains queued.

Level-triggered code can therefore look like this conceptually:

if (events[i].events & EPOLLIN) {
    ssize_t count = recv(fd, buffer, sizeof buffer, 0);
    if (count > 0) {
        handle_bytes(buffer, count);
    } else if (count == 0) {
        close(fd);              // peer performed an orderly shutdown
    } else if (errno != EAGAIN && errno != EWOULDBLOCK) {
        close(fd);              // handle the actual error
    }
}

The descriptor should still be nonblocking: readiness is a useful indication, not an unconditional guarantee that a subsequent operation cannot block. Another thread, handler, or operation can consume the data first.

Edge-triggered behavior

Passing EPOLLET requests edge-triggered (ET) behavior. Instead of repeatedly reporting the current ready state, epoll primarily reports a transition—for example, a descriptor changing from not readable to readable.

That changes the handler's responsibility. If data is available and the handler reads only part of it, the descriptor may remain readable without producing another edge. The event loop can then wait indefinitely even though unread data is still present.

With ET, the usual pattern is to use nonblocking descriptors and drain the operation until it would block:

for (;;) {
    ssize_t count = recv(fd, buffer, sizeof buffer, 0);

    if (count > 0) {
        handle_bytes(buffer, count);
        continue;
    }
    if (count == 0) {
        close(fd);              // EOF
        break;
    }
    if (errno == EAGAIN || errno == EWOULDBLOCK) {
        break;                  // fully drained for now
    }
    close(fd);                  // other error
    break;
}

The same rule applies to accepting connections: after an EPOLLIN event on a listening socket, repeatedly call nonblocking accept until it returns EAGAIN or EWOULDBLOCK. For writable sockets, keep sending queued output until the queue is empty or send would block; then stop watching for EPOLLOUT until more output exists.

ET can reduce repeated notifications and make high-throughput loops efficient, but it makes incomplete consumption a correctness bug rather than merely a performance issue. LT is often easier to implement; ET is useful when the application is prepared to drain each readiness condition carefully.

What logs and flags are telling you

A stack trace mentioning epoll_wait, or a configuration flag such as EPOLLET, is describing the event loop's waiting strategy—not a separate kind of socket. The socket remains a normal nonblocking file descriptor. Epoll only reports when its requested operations appear able to proceed.

In practice, the key question is: after receiving a notification, does the handler consume the descriptor until it would block, or can it safely leave work for the next wait? The first is required for edge-triggered operation; the second is supported by level-triggered operation.

← All articles