Edmondo's Vault

Multi-Threaded epoll HTTP File Server Architecture

Jul 07, 202616 min read

The program is an HTTP static-file server where multiple socket-worker threads use epoll. The note follows one accepted client socket through accept, worker handoff, epoll registration, request read, response write, and the point where blocking disk reads leave the socket worker.

Prerequisites

Read Serving Files Over HTTP with Linux I-O Models, epoll, and eventfd first.

Program Roles

The server has three kinds of threads:

RoleCountOwnsReceivesProduces
Acceptor thread1listen_fdNew TCP connections from the kernel.Accepted client sockets, named conn_fd.
Socket worker threadMAX_WORKERSOne epoll instance, one eventfd, and its assigned client connections.Queued conn_fd values and socket readiness events.Request parsing progress, response writes, and optional file-read jobs.
Disk worker threadOptional poolBlocking open / read work for static files.File-read jobs from socket workers.Completed file-read jobs for the owning socket worker.

MAX_WORKERS = 1 gives one socket event loop. MAX_WORKERS = 4 gives four independent socket event loops. The mechanism is the same; only the number of worker threads changes.

One Request Path

The diagram shows the handoff boundaries. enqueue_accepted_socket only moves conn_fd into the worker inbox. eventfd_write wakes the worker. register_new_connections is where the worker allocates struct conn and registers conn_fd in epoll.

One HTTP request moves through these steps:

StepThreadFunctionState being moved
1Acceptoraccept_loopaccept returns conn_fd, a connected client socket.
2Acceptorenqueue_accepted_socketconn_fd is placed in one worker’s accepted_sockets inbox.
3Acceptoreventfd_writeThe worker’s wake_fd becomes readable.
4Socket workerworker_mainepoll_wait returns an ITEM_WAKE event.
5Socket workerregister_new_connectionsconn_fd becomes struct conn, then gets registered in epoll.
6Socket workeron_readableRequest bytes accumulate in c->req.
7Socket workerstart_responseResponse bytes are prepared, or a disk job is submitted.
8Socket workeron_writableResponse bytes are written from c->out to c->fd.

Full Skeleton

Functions grouped by the thread that calls them:

// epoll_file_server.c
 
#define MAX_WORKERS 4
 
int main(void);  // create listen_fd, start workers, enter accept loop
 
// Acceptor thread.
static void start_workers(struct worker workers[MAX_WORKERS]);  // initialize each worker and create its thread
static void accept_loop(int listen_fd,
                        struct worker workers[MAX_WORKERS]);  // accept client sockets and hand them to workers
 
// Socket worker thread.
static void *worker_main(void *arg);  // wait in epoll_wait and dispatch ready items
static void register_wake_fd(struct worker *w);  // add w->wake_fd to w->epfd
static void handle_ready_item(struct worker *w,
                              struct epoll_event *ev);  // route wake events and socket events
 
// Accepted-socket inbox for a socket worker.
static void accepted_socket_queue_init(struct accepted_socket_queue *q);  // initialize an empty queue of conn_fd values
static void enqueue_accepted_socket(struct worker *w,
                       int conn_fd);  // queue one accepted client socket for this worker
static int dequeue_accepted_socket(struct worker *w,
                      int *conn_fd_out);  // pop one accepted client socket, or return 0
static void register_new_connections(struct worker *w);  // turn queued conn_fd values into struct conn objects
 
// One client connection, owned by one socket worker.
static void handle_connection_event(struct worker *w,
                                    struct conn *c,
                                    uint32_t events);  // call read/write handlers for one connection
static int on_readable(struct worker *w,
                       struct conn *c);  // read request bytes until complete or EAGAIN
static void start_response(struct worker *w,
                           struct conn *c);  // prepare output and switch interest to EPOLLOUT
static int on_writable(struct worker *w,
                       struct conn *c);  // write response bytes until done or EAGAIN
static int close_conn(struct worker *w,
                      struct conn *c);  // remove fd from epoll, close it, free state
 
// Optional disk-worker path.
static void *disk_worker_main(void *arg);  // run blocking file reads outside socket workers
static void submit_file_job(struct file_job *job);  // queue a file read request for disk workers
static void completed_file_job_queue_init(struct completed_file_job_queue *q);  // initialize an empty queue of completed file jobs
static void enqueue_completed_file_job(struct worker *w,
                                       struct file_job *job);  // return a completed file job to its socket worker
static int dequeue_completed_file_job(struct worker *w,
                                      struct file_job **job_out);  // pop one completed file job, or return 0
static void process_completed_file_jobs(struct worker *w);  // copy file results into conn state and arm EPOLLOUT

The blocking server has one large function:

serve_connection(conn_fd);

The epoll server splits that function across event handlers:

Blocking stepEvent-driven replacement
Read request bytes.on_readable runs when epoll_wait reports EPOLLIN for conn_fd.
Decide the response.start_response runs after request headers are complete.
Write response bytes.on_writable runs when epoll_wait reports EPOLLOUT for conn_fd.
Close connection.close_conn removes conn_fd from epoll, closes it, and frees struct conn.

Data Model

File Descriptors

In Linux C code, fd means file descriptor: an integer handle returned by a kernel API. A socket descriptor, an epoll instance descriptor, an eventfd descriptor, and a disk-file descriptor are all integers. The API that created the descriptor determines what the program may do with it.

NameCreated byMeaningRegistered in epoll?
listen_fdlisten_socketListening TCP socket. The acceptor calls accept(listen_fd, ...) on it.No in this design.
conn_fd / c->fdacceptConnected TCP socket for one client.Yes. Socket workers watch it for EPOLLIN and EPOLLOUT.
w->epfdepoll_create1The worker’s epoll instance.No. The worker passes it to epoll_ctl and epoll_wait.
w->wake_fdeventfdWakeup descriptor for cross-thread notifications.Yes. A write makes it readable.
file_fd / local fdopenRegular disk file.No. Regular file reads are not made asynchronous by epoll.

EPOLLIN And EPOLLOUT

EPOLLIN and EPOLLOUT are readiness flags for a watched descriptor. They tell the worker which operation is likely to make progress on that descriptor without blocking.

FlagMeaning for conn_fdHandler in this server
EPOLLINThe client socket has bytes to read, or the peer closed the readable side.on_readable calls read(c->fd, ...).
EPOLLOUTThe client socket has buffer space for more outgoing bytes.on_writable calls write(c->fd, ...).

The server registers a new client socket with EPOLLIN because the first thing it expects from a client is an HTTP request. After the request is parsed and the response bytes are ready, start_response changes the interest to EPOLLOUT because the server now wants to write the response.

An epoll_event has two separate parts:

FieldSet byMeaning
ev.events during epoll_ctlApplicationInterest mask: which readiness changes the worker wants to hear about.
events[i].events returned by epoll_waitKernelReady mask: which readiness flags are true now.
ev.data.ptr during epoll_ctlApplicationUser payload copied back unchanged by epoll_wait.
events[i].data.ptr returned by epoll_waitKernel copies application valuePointer used to recover the worker’s ITEM_WAKE or ITEM_CONN state.

For a new client socket, the application registers interest in read readiness:

struct epoll_event ev = {
    .events = EPOLLIN,
    .data.ptr = &c->item,
};
epoll_ctl(w->epfd, EPOLL_CTL_ADD, conn_fd, &ev);

Later, when epoll_wait returns, the worker reads both fields:

struct epoll_item *item = events[i].data.ptr;
struct conn *c = item->ptr;
 
if (events[i].events & EPOLLIN)
    on_readable(w, c);

So EPOLLIN is not the payload. EPOLLIN is in the returned readiness mask. The payload is data.ptr, which points back to the connection item registered with epoll_ctl.

Worker State

One socket worker owns one struct worker.

enum item_kind {
    ITEM_WAKE,
    ITEM_CONN,
};
 
struct epoll_item {
    enum item_kind kind;
    void *ptr;
};
 
struct worker {
    int epfd;        // fd returned by epoll_create1
    int wake_fd;     // fd returned by eventfd
    struct epoll_item wake_item;
 
    struct accepted_socket_queue accepted_sockets;
    struct completed_file_job_queue completed_file_jobs;
    pthread_mutex_t mu;
 
    pthread_t thread;
};

w->epfd is the worker’s epoll instance. It stores the worker’s watched descriptors and ready events inside the kernel. The program gets it from epoll_create1, then passes it to epoll_ctl to change the watched set and to epoll_wait to sleep until one watched descriptor becomes ready. Shutdown code may eventually call close(w->epfd), but that is cleanup, not the reason the field exists.

w->wake_fd is an eventfd descriptor registered inside w->epfd. Producer threads write to w->wake_fd after they put work in one of the worker’s queues. In this server, the producers are the acceptor thread and the optional disk worker threads.

Connection State

One accepted client socket becomes one struct conn.

struct conn {
    struct epoll_item item;
    int fd;              // same socket as conn_fd
 
    char req[4096];
    size_t req_len;
 
    char out[16384];
    size_t out_len;
    size_t out_sent;
};

Only the owning socket worker mutates a connection’s struct conn. The ownership rule keeps the per-connection buffers and offsets sane.

Worker Inboxes

The worker has two project-local queues. They are not Linux APIs.

Queue fieldContainsProducerConsumer
w->accepted_socketsAccepted client socket descriptors (conn_fd)Acceptor threadOwning socket worker
w->completed_file_jobsFinished disk-read jobsDisk worker threadsOwning socket worker

accepted_socket_queue carries only accepted client socket descriptors. It does not contain listen_fd, w->epfd, w->wake_fd, or disk file_fd values.

The queues are inboxes because producer threads need to hand work to a sleeping socket worker. The producer cannot just write into the worker’s current stack frame. It puts an item in a queue protected by w->mu, then writes to w->wake_fd. The worker wakes from epoll_wait, drains the queues, and performs the state mutation in its own thread.

enqueue_accepted_socket(w, conn_fd) copies the integer socket descriptor into w->accepted_sockets. It does not read from the socket, write to the socket, allocate struct conn, or register anything in epoll.

static void enqueue_accepted_socket(struct worker *w, int conn_fd) {
    pthread_mutex_lock(&w->mu);
    accepted_socket_queue_push(&w->accepted_sockets, conn_fd);
    pthread_mutex_unlock(&w->mu);
}

The worker later removes that same integer with dequeue_accepted_socket(w, &conn_fd). Only then does the worker allocate struct conn and call epoll_ctl(w->epfd, EPOLL_CTL_ADD, conn_fd, &ev).

The initialization calls create empty inboxes:

accepted_socket_queue_init(&w->accepted_sockets);
completed_file_job_queue_init(&w->completed_file_jobs);

Those functions set queue bookkeeping to an empty state. In a ring-buffer implementation, that means head = 0, tail = 0, and len = 0. In a linked-list implementation, that means head = NULL and tail = NULL. Without initialization, the first enqueue or dequeue would read uninitialized memory.

completed_file_job_queue_init appears even if the first version of the server does not use disk workers yet. It initializes the inbox that the optional disk-worker path uses later in the note.

Startup

main creates the listening socket, starts socket workers, then becomes the acceptor thread.

int main(void) {
    int listen_fd = listen_socket(8080);
    struct worker workers[MAX_WORKERS];
 
    start_workers(workers);
    accept_loop(listen_fd, workers);
}

start_workers must initialize each worker before pthread_create, because the new thread may start running immediately.

static void start_workers(struct worker workers[MAX_WORKERS]) {
    for (size_t i = 0; i < MAX_WORKERS; i++) {
        struct worker *w = &workers[i];
 
        w->epfd = epoll_create1(0);
        if (w->epfd < 0)
            die("epoll_create1");
 
        w->wake_fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
        if (w->wake_fd < 0)
            die("eventfd");
 
        accepted_socket_queue_init(&w->accepted_sockets);
        completed_file_job_queue_init(&w->completed_file_jobs);
        pthread_mutex_init(&w->mu, NULL);
 
        pthread_create(&w->thread, NULL, worker_main, w);
    }
}

After start_workers, every worker has:

FieldUsed for
w->epfdWaiting for socket readiness and wakeup readiness.
w->wake_fdLetting other threads wake this worker.
w->accepted_socketsReceiving newly accepted client sockets.
w->completed_file_jobsReceiving completed file-read results.
w->muProtecting queue mutation.

Flow 1: Accept A Client Socket

The acceptor accepts a TCP connection and gets a connected client socket descriptor, conn_fd.

static void accept_loop(int listen_fd, struct worker workers[MAX_WORKERS]) {
    size_t next = 0;
 
    for (;;) {
        int conn_fd = accept(listen_fd, NULL, NULL);
        if (conn_fd < 0) {
            if (errno == EINTR)
                continue;
            die("accept");
        }
 
        set_nonblocking(conn_fd);
 
        struct worker *w = &workers[next++ % MAX_WORKERS];
        enqueue_accepted_socket(w, conn_fd);
        eventfd_write(w->wake_fd, 1);
    }
}

The last two lines are the handoff:

enqueue_accepted_socket(w, conn_fd);
eventfd_write(w->wake_fd, 1);

enqueue_accepted_socket puts the accepted client socket into the worker’s accepted_sockets inbox. eventfd_write wakes the worker so it notices that inbox work exists.

Flow 2: Wake The Socket Worker

Each worker starts by registering its wakeup descriptor in its own epoll instance. After this call, w->epfd watches w->wake_fd.

static void register_wake_fd(struct worker *w) {
    w->wake_item.kind = ITEM_WAKE;
    w->wake_item.ptr = w;
 
    struct epoll_event ev = {
        .events = EPOLLIN,
        .data.ptr = &w->wake_item,
    };
    epoll_ctl(w->epfd, EPOLL_CTL_ADD, w->wake_fd, &ev);
}
 
static void *worker_main(void *arg) {
    struct worker *w = arg;
    register_wake_fd(w);
 
    for (;;) {
        struct epoll_event events[64];
        int n = epoll_wait(w->epfd, events, 64, -1);
 
        for (int i = 0; i < n; i++)
            handle_ready_item(w, &events[i]);
    }
}

epoll_wait can return either:

Ready itemMeaning
ITEM_WAKEThe acceptor or a disk worker queued work for this worker and wrote to w->wake_fd.
ITEM_CONNA client socket owned by this worker became readable, writable, hung up, or errored.
static void handle_ready_item(struct worker *w, struct epoll_event *ev) {
    struct epoll_item *item = ev->data.ptr;
 
    if (item->kind == ITEM_WAKE) {
        drain_wake_fd(w);
        register_new_connections(w);
        process_completed_file_jobs(w);
        return;
    }
 
    struct conn *c = item->ptr;
    handle_connection_event(w, c, ev->events);
}

drain_wake_fd consumes the eventfd counter. The counter is not the work queue; the queues are the source of truth.

static void drain_wake_fd(struct worker *w) {
    uint64_t value;
 
    for (;;) {
        int rc = eventfd_read(w->wake_fd, &value);
        if (rc == 0)
            continue;
 
        if (errno == EAGAIN)
            return;
 
        die("eventfd_read");
    }
}

Flow 3: Register The Client Socket

When the worker drains accepted_sockets, each queued conn_fd becomes a worker-owned struct conn.

static void register_new_connections(struct worker *w) {
    int conn_fd;
 
    while (dequeue_accepted_socket(w, &conn_fd)) {
        struct conn *c = calloc(1, sizeof(*c));
        c->item.kind = ITEM_CONN;
        c->item.ptr = c;
        c->fd = conn_fd;
 
        struct epoll_event ev = {
            .events = EPOLLIN,
            .data.ptr = &c->item,
        };
        epoll_ctl(w->epfd, EPOLL_CTL_ADD, conn_fd, &ev);
    }
}

After epoll_ctl(... EPOLL_CTL_ADD ...), the worker’s epoll_wait can report EPOLLIN when the client sends request bytes on that socket.

Flow 4: Read, Prepare, Write

Keep three storage locations separate:

DataStored inFilled byConsumed by
HTTP request bytesc->reqon_readable reads from c->fd.start_response parses the request.
File bytes from diskjob->resultdisk_worker_main reads from file_fd.process_completed_file_jobs copies into c->out.
Response bytes to writec->outstart_response or process_completed_file_jobs.on_writable writes to c->fd.
epoll payloadev.data.ptr / events[i].data.ptrRegistration code stores &c->item.Dispatcher recovers struct conn *c.

The epoll_event does not carry file bytes. It carries readiness flags in events and a pointer to connection state in data.ptr.

The connection dispatcher routes readiness flags for one worker-owned connection.

static void handle_connection_event(struct worker *w,
                                    struct conn *c,
                                    uint32_t events) {
    if (events & (EPOLLERR | EPOLLHUP)) {
        close_conn(w, c);
        return;
    }
 
    if (events & EPOLLIN) {
        if (on_readable(w, c) < 0)
            return;
    }
 
    if (events & EPOLLOUT) {
        if (on_writable(w, c) < 0)
            return;
    }
}

on_readable reads bytes from the client socket into c->req. EAGAIN means the non-blocking socket has no more bytes right now, so the worker returns to epoll_wait.

static int on_readable(struct worker *w, struct conn *c) {
    for (;;) {
        ssize_t n = read(c->fd,
                         c->req + c->req_len,
                         sizeof(c->req) - c->req_len);
 
        if (n < 0 && errno == EAGAIN)
            return 0;
        if (n <= 0)
            return close_conn(w, c);
 
        c->req_len += (size_t)n;
 
        if (headers_complete(c)) {
            start_response(w, c);
            return 0;
        }
 
        if (c->req_len == sizeof(c->req))
            return close_conn(w, c);
    }
}

Without disk workers, start_response does the response preparation directly in the socket worker. on_readable calls it after the HTTP request headers are complete. start_response fills c->out, resets c->out_sent, then changes the socket interest from read readiness to write readiness.

static void start_response(struct worker *w, struct conn *c) {
    c->out_len = build_response(c->req, c->req_len,
                                c->out, sizeof(c->out));
    c->out_sent = 0;
 
    struct epoll_event ev = {
        .events = EPOLLOUT,
        .data.ptr = &c->item,
    };
    epoll_ctl(w->epfd, EPOLL_CTL_MOD, c->fd, &ev);
}

on_writable assumes response bytes are already in c->out before the socket is armed for EPOLLOUT.

PathWho fills c->outWhen EPOLLOUT is armed
No disk-worker pathstart_response writes headers/body into c->out and sets c->out_len.Immediately after start_response fills c->out.
Disk-worker pathprocess_completed_file_jobs copies job->result into c->out and sets c->out_len.After the disk worker returns the completed file_job.

on_writable does not fetch data from epoll. It receives struct conn *c, reads the already-filled c->out buffer, and writes those bytes to c->fd. Partial writes are normal, so c->out_sent records how much of the response already reached the socket.

static int on_writable(struct worker *w, struct conn *c) {
    while (c->out_sent < c->out_len) {
        ssize_t n = write(c->fd,
                          c->out + c->out_sent,
                          c->out_len - c->out_sent);
 
        if (n < 0 && errno == EAGAIN)
            return 0;
        if (n <= 0)
            return close_conn(w, c);
 
        c->out_sent += (size_t)n;
    }
 
    return close_conn(w, c);
}

Static Files And Disk Workers

The socket worker should not read regular files directly:

static void start_response(struct worker *w, struct conn *c) {
    int file_fd = open(path, O_RDONLY);
    ssize_t n = read(file_fd, c->out, sizeof(c->out));
    c->out_len = (size_t)n;
    switch_to_epollout(w, c);
}

epoll can report readiness for sockets. It does not turn regular disk-file reads into asynchronous operations. If open or read blocks, that socket worker stops advancing every other connection it owns.

A disk worker pool moves blocking file I/O away from socket workers. The socket worker still owns struct conn; the disk worker owns the blocking open / read.

The handoff object is struct file_job *. In this teaching version, file_job contains the byte buffer:

FieldWritten byRead byMeaning
job->pathSocket workerDisk workerFile path to open.
job->resultDisk workerSocket workerFile bytes read from disk.
job->result_lenDisk workerSocket workerNumber of valid bytes in result.
job->connSocket workerSocket worker after completionConnection waiting for these bytes.

The disk worker returns file bytes to the socket worker by passing back a pointer to a job object that contains the bytes. The socket worker later copies those bytes into c->out and writes them to the client socket.

struct file_job {
    struct worker *owner;
    struct conn *conn;
    char path[1024];
    char result[16384];
    size_t result_len;
    int status;
};
 
static void start_response(struct worker *w, struct conn *c) {
    struct file_job *job = make_file_job(w, c);
    submit_file_job(job);
}

The disk worker fills the job result and returns it to the owning socket worker.

static void *disk_worker_main(void *arg) {
    for (;;) {
        struct file_job *job = take_file_job();
 
        int fd = open(job->path, O_RDONLY);
        if (fd < 0) {
            job->status = -1;
        } else {
            ssize_t n = read(fd, job->result, sizeof(job->result));
            close(fd);
 
            job->status = n < 0 ? -1 : 0;
            job->result_len = n > 0 ? (size_t)n : 0;
        }
 
        enqueue_completed_file_job(job->owner, job);
        eventfd_write(job->owner->wake_fd, 1);
    }
}

The socket worker later drains completed_file_jobs, copies the result into connection state, and switches the client socket to EPOLLOUT.

static void process_completed_file_jobs(struct worker *w) {
    struct file_job *job;
 
    while (dequeue_completed_file_job(w, &job)) {
        struct conn *c = job->conn;
 
        if (job->status < 0)
            c->out_len = build_404(c->out, sizeof(c->out));
        else {
            memcpy(c->out, job->result, job->result_len);
            c->out_len = job->result_len;
        }
        c->out_sent = 0;
 
        struct epoll_event ev = {
            .events = EPOLLOUT,
            .data.ptr = &c->item,
        };
        epoll_ctl(w->epfd, EPOLL_CTL_MOD, c->fd, &ev);
        free(job);
    }
}

The EPOLLOUT registration does not attach bytes to the epoll event. It tells epoll to wake the socket worker when c->fd can accept outgoing bytes. The bytes stay in c->out; data.ptr points back to c->item, so the worker can find c->out when on_writable runs.

While a disk job is in flight, the socket worker must not free c while the disk worker still has a pointer to it. Production code often uses reference counts, cancellation flags, or message passing with opaque connection IDs instead of raw pointers.

Shared epoll Variant

The design above gives each worker its own epoll instance. Linux also allows multiple threads to wait on the same epoll instance:

int n = epoll_wait(shared_epfd, events, 64, -1);

Shared waiting does not make connection state safe. If two workers can run handlers for the same struct conn, request length, output offsets, close state, and buffer ownership all need synchronization.

EPOLLONESHOT is one ownership protocol for a shared epoll instance. It disables a file descriptor after one event is delivered.

struct epoll_event ev = {
    .events = EPOLLIN | EPOLLONESHOT,
    .data.ptr = &c->item,
};
epoll_ctl(epfd, EPOLL_CTL_ADD, c->fd, &ev);

The worker that receives the event owns the connection for that step. When it finishes, it rearms the file descriptor:

ev.events = EPOLLOUT | EPOLLONESHOT;
ev.data.ptr = &c->item;
epoll_ctl(epfd, EPOLL_CTL_MOD, c->fd, &ev);

In the per-worker design used by most of this note, one connection belongs to one event loop, so the connection state does not need per-field locking.

See also

Sources

  • Linux man-pages: epoll(7), epoll_ctl(2), and eventfd(2).

Graph View

Backlinks