Serving Files Over HTTP with Linux I-O Models

A static HTTP file server waits in two different places:

• socket I/O: the client connects, sends request bytes, and receives response bytes at its own pace
• file I/O: the server opens and reads a file, which may wait on filesystem metadata, the page cache, or storage

Background notes

Keep IO Selectors, epoll, io_uring, and Threads nearby if the API names or blocking-thread baseline are unfamiliar.

Recommended path

1. Read this map first.
2. Read eventfd before the multi-threaded epoll server note.
3. Read Multi-Threaded epoll HTTP File Server Architecture to see how serve(conn_fd) is split into readiness handlers.
4. Read io_uring Server Architecture to see how serve(conn_fd) is split into submissions and completions.

Blocking Baseline

In the blocking version, the server gives one accepted connection to one function. serve_connection handles the request from first byte to close.

int main(void) {
    int listen_fd = listen_socket(8080);
 
    for (;;) {
        int conn_fd = accept(listen_fd, NULL, NULL);
        spawn_thread(serve_connection, conn_fd);
    }
}
 
static void *serve_connection(void *arg) {
    int conn_fd = (int)(intptr_t)arg;
 
    read_http_request(conn_fd);
    open_requested_file();
    write_http_response(conn_fd);
    close(conn_fd);
}

The call stack is the state machine. The thread blocks wherever the next step blocks.

epoll and io_uring both replace the blocking serve_connection body with explicit state stored outside the call stack.

Three Ways To Split serve_connection

Model	What replaces serve_connection	What wakes the program	Where request state lives
Thread per connection	Nothing. The function keeps the request lifecycle in one call stack.	Blocking syscalls return.	The thread stack.
epoll	Readiness handlers such as on_readable and on_writable.	epoll_wait says a socket can make progress.	A heap struct conn owned by one worker.
io_uring	Submission/completion pairs such as submit_recv and on_recv.	A completion queue event (CQE) says an operation finished.	A heap struct conn reached through CQE user_data.

The I/O API does not choose the threading architecture. A server can run one worker, many workers, one ring per worker, or one shared ring; each design still needs a rule for which thread may mutate a connection’s state.

One Request Through Each Model

Blocking Thread

The request stays inside one function.

serve_connection(conn_fd)
    read request bytes
    open file
    read file bytes
    write response bytes
    close connection

If file reading blocks, only this connection’s thread is blocked. Many concurrent connections mean many threads.

epoll Worker

The request is resumed by readiness events.

accept_loop
    accept conn_fd
    assign conn_fd to worker
 
worker_main
    epoll_wait
    on_readable(c)      // read request bytes until EAGAIN or headers complete
    start_response(c)   // prepare response or hand file work elsewhere
    on_writable(c)      // write response bytes until EAGAIN or finished

One worker runs one handler at a time. It still handles many connections because most connections are waiting most of the time. If a handler performs a blocking file read, every other connection owned by that worker waits behind it.

io_uring Worker

The request is resumed by operation completions.

accept_loop
    accept conn_fd
    assign conn_fd to worker
 
worker_main
    submit_recv(c)
    wait for CQE
    on_recv(c)
    submit_open_file(c)
    wait for CQE
    on_open_file(c)
    submit_send_headers(c)
    wait for CQE
    on_send_headers(c)
    submit_file_read(c)
    wait for CQE
    on_read_file(c)
    submit_send_body(c)
    wait for CQE
    on_send_body(c)

The worker does not call read(file_fd, buf, len) directly. It submits a file read and returns to the completion loop. Other completions can be processed while the file operation is outstanding.

Why File Reads Change The Design

Socket readiness fits epoll well:

• the listening socket becomes readable when accept can make progress
• a client socket becomes readable when request bytes are queued
• a client socket becomes writable when send-buffer space exists

Regular file reads do not fit the same model. Linux epoll_ctl rejects regular disk files, and epoll does not report “this file read completed.” An epoll file server that must avoid blocking the socket worker needs another mechanism:

Addition	Why it exists
Disk worker pool	Move blocking open and file read out of socket workers.
sendfile	The kernel copies file bytes to the socket for simple static-file responses.
io_uring	Submit both socket and file operations and react to completions.

Review Checks

After reading the three notes, check that you can explain:

• what serve_connection looks like in the blocking model
• how epoll splits serve_connection into readiness handlers
• how io_uring splits serve_connection into submit/completion pairs
• why one event loop is concurrent but not parallel
• why each connection needs one clear owner
• why regular file reads are the awkward part of an epoll file server

See also

• Multi-Threaded epoll HTTP File Server Architecture
• io_uring Server Architecture
• IO Selectors
• epoll
• eventfd
• io_uring
• Threads
• OS Page Cache

Sources

• Phil Eaton, “Serving files over HTTP three ways: synchronous, epoll, and io_uring”, The Consensus.
• Linux man-pages: epoll(7), epoll_ctl(2), eventfd(2), io_uring(7), and io_uring_setup(2).