arthrop0d

Brief articles on varied topics.

Linux epoll: Interest Lists, Ready Lists, and Triggers

operating systemsnetworkingconcurrency

Linux epoll is a kernel mechanism that lets one thread wait efficiently for activity on many file descriptors, such as sockets, pipes, and event objects. It does this by keeping an interest list of descriptors you want to watch and a ready list of descriptors that currently have an event to report.

The problem epoll solves

A server may have thousands of client sockets but only a few that can make progress at any moment. A simple design could dedicate one thread to each connection, but that consumes memory and scheduling time. Another design could repeatedly inspect every socket, asking whether each can be read or written. That wastes work when most sockets are idle.

Traditional interfaces such as select and poll represent the second approach: the program passes a set of descriptors to the kernel, and the kernel checks the set during each wait. The program then scans the returned set. As the number of mostly idle connections grows, repeatedly checking all of them becomes expensive.

epoll lets the kernel remember the set being watched. When a descriptor becomes ready, the kernel records that fact, so a wait can return the descriptors with work rather than making the application rescan every descriptor.

The interest list

An epoll instance is created with epoll_create1. The application then adds descriptors to its interest list with epoll_ctl. Each registration says which events matter, such as:

The registration can also carry application data, commonly a file descriptor or a pointer to a connection object.

int epfd = epoll_create1(0);

struct epoll_event watch = {
    .events = EPOLLIN | EPOLLET,
    .data.fd = client_fd
};

if (epoll_ctl(epfd, EPOLL_CTL_ADD, client_fd, &watch) == -1) {
    /* handle the error */
}

Conceptually, this registration is the descriptor’s entry on the interest list: “tell me when this descriptor is readable, using edge-triggered notifications.” The application can later change or remove it with EPOLL_CTL_MOD or EPOLL_CTL_DEL.

The ready list and epoll_wait

The kernel monitors registered descriptors using their normal readiness rules. For a socket, for example, incoming bytes can make it readable. Space becoming available in a send buffer can make it writable. When a watched descriptor becomes ready, epoll places it on its ready list, or records that it is ready if it is already there.

The application calls epoll_wait to sleep until one or more entries are ready:

struct epoll_event events[64];

int count = epoll_wait(epfd, events, 64, -1);
for (int i = 0; i < count; i++) {
    int fd = events[i].data.fd;
    /* service fd */
}

The call returns the ready entries, up to the size of the supplied array. If none are ready, the thread sleeps inside the kernel rather than polling in a loop. When activity occurs, the kernel wakes it and returns the descriptors that need attention.

This is the central efficiency gain: the cost of a wait is related mainly to the descriptors that became ready, not to every descriptor registered with the epoll instance. The application still has to process the returned descriptors, so handling thousands of simultaneous events can still be expensive; epoll removes the unnecessary scan of thousands of idle ones.

Level-triggered notification

Level-triggered mode is the default. It reports a descriptor whenever the condition remains true. If a socket has unread data, epoll_wait reports it. If the application reads only some of that data and waits again, the socket is still readable, so it can be reported again.

This resembles repeatedly asking, “is this descriptor ready right now?” It is usually easier to implement because work can be handled in smaller pieces:

  1. Wait for EPOLLIN.
  2. Read some data.
  3. Return to epoll_wait.
  4. Receive another notification if data remains.

Level-triggered mode is forgiving when a handler does not drain a descriptor completely. It may produce repeated notifications, but those notifications reflect unfinished work.

Edge-triggered notification

Edge-triggered mode reports a change in readiness rather than continuously reporting the current condition. If a socket changes from having no readable data to having readable data, epoll reports that transition. If the application reads only part of the data, the socket may remain readable without another edge occurring.

That means an edge-triggered handler must normally use a nonblocking descriptor and keep reading or writing until the operation returns EAGAIN or EWOULDBLOCK:

for (;;) {
    ssize_t n = read(fd, buffer, sizeof buffer);

    if (n > 0) {
        handle_bytes(buffer, n);
    } else if (n == -1 && (errno == EAGAIN || errno == EWOULDBLOCK)) {
        break;              /* drained for now */
    } else {
        close_connection(fd); /* EOF or another error */
        break;
    }
}

The same principle applies to writing: keep sending while possible, then wait for another writable notification when the send buffer fills. Failing to drain a descriptor is a common edge-triggered bug: the application leaves data behind but receives no new edge, so the connection appears to stall.

Choosing between them

Level-triggered epoll is often the safer default. Edge-triggered mode can reduce repeated notifications and is useful when handlers already drain nonblocking descriptors, but it demands stricter event-loop code.

In both modes, readiness is only a statement that an operation was likely not to block when the kernel checked it. Another thread, handler, or process can consume the data first, and errors or connection closure can change the result. For that reason, event-driven code should handle short reads, EOF, EAGAIN, and other errors rather than assuming one notification equals one complete request.

If a log or configuration mentions epoll, EPOLLET, or an “epoll wait,” it is referring to this arrangement: an application-maintained interest list, a kernel-maintained set of ready notifications, and one or more calls that sleep until useful work appears.

← All articles