io_uring Server Architecture

An io_uring HTTP server replaces the blocking serve_connection(conn_fd) function with a chain of submitted operations and completion handlers. The worker submits work to the kernel, waits for completion queue events (CQEs), and uses each CQE to resume the connection state.

Prerequisites

Read Serving Files Over HTTP with Linux I-O Models and io_uring first.

Full Program Skeleton

Program outline:

// uring_file_server.c
 
#define MAX_WORKERS 4
 
int main(void);
 
static void start_workers(struct worker workers[MAX_WORKERS]);
static void accept_loop(int listen_fd, struct worker workers[MAX_WORKERS]);
 
static void *worker_main(void *arg);
static void register_new_connections(struct worker *w);
static void dispatch_completion(struct worker *w,
                                struct op *op,
                                struct io_uring_cqe *cqe);
 
static void submit_wake_read(struct worker *w);
static void on_wake(struct worker *w);
 
static void submit_recv(struct worker *w, struct conn *c);
static void on_recv(struct worker *w, struct conn *c, int n);
 
static void start_response(struct worker *w, struct conn *c);
static void submit_open_file(struct worker *w, struct conn *c);
static void on_open_file(struct worker *w, struct conn *c, int fd);
 
static void submit_send_headers(struct worker *w, struct conn *c);
static void on_send_headers(struct worker *w, struct conn *c, int n);
 
static void submit_file_read(struct worker *w, struct conn *c);
static void on_read_file(struct worker *w, struct conn *c, int n);
 
static void submit_send_body(struct worker *w, struct conn *c);
static void on_send_body(struct worker *w, struct conn *c, int n);
 
static void submit_close(struct worker *w, struct conn *c);
static void on_close(struct worker *w, struct conn *c);

serve_connection(conn_fd) becomes submit/completion pairs:

Pair	Meaning
`submit_recv` / `on_recv`	Ask for request bytes, then resume when bytes arrive.
`submit_open_file` / `on_open_file`	Ask to open the requested file, then resume with a file descriptor or error.
`submit_file_read` / `on_read_file`	Ask for file bytes, then resume with a body chunk.
`submit_send_body` / `on_send_body`	Ask to send bytes, then resume when some bytes were accepted.
`submit_close` / `on_close`	Ask to close the socket, then free connection state.

State And Ownership

enum op_kind {
    OP_WAKE,
    OP_RECV,
    OP_OPEN_FILE,
    OP_SEND_HEADERS,
    OP_READ_FILE,
    OP_SEND_BODY,
    OP_CLOSE,
};
 
struct op {
    enum op_kind kind;
    struct conn *conn;
};
 
struct worker {
    struct io_uring ring;
    int wake_fd;             // fd returned by eventfd
    uint64_t wake_value;
    struct fd_queue new_fds;
    pthread_mutex_t mu;
    pthread_t thread;
};
 
struct conn {
    int fd;
    int file_fd;
    char req[4096];
    size_t req_len;
    char path[1024];
    char headers[512];
    size_t headers_len;
    size_t headers_sent;
    char body[16384];
    size_t body_len;
    size_t body_sent;
    int close_after_headers;
};

Ownership matches the epoll worker design:

the acceptor assigns a new connection to one worker
each worker owns one io_uring ring
the owning worker submits submission queue entries (SQEs) for that connection
the owning worker consumes completion queue events (CQEs) for that ring
only that worker mutates the connection’s struct conn

The baseline keeps one in-flight operation per connection. Production servers may pipeline more operations; pipelining needs stricter ownership and lifetime rules.

struct op is the bridge between a submitted operation and the completion handler. The application stores struct op * in SQE user_data; the kernel copies it back into the CQE.

Core io_uring Calls

The server uses these liburing calls:

io_uring_queue_init(256, &w->ring, 0);
 
struct io_uring_cqe *cqe;
unsigned head, count = 0;
 
io_uring_submit_and_wait(&w->ring, 1);
io_uring_for_each_cqe(&w->ring, head, cqe) {
    dispatch_completion(w, io_uring_cqe_get_data(cqe), cqe);
    count++;
}
io_uring_cq_advance(&w->ring, count);

The worker creates a ring, submits queued operations, waits for at least one completion queue event (CQE), iterates over completed operations, and tells the kernel how many CQEs it consumed.

struct io_uring_sqe *sqe = io_uring_get_sqe(&w->ring);
io_uring_sqe_set_data(sqe, op);

Every submitted operation starts by taking a submission queue entry (SQE) from the ring. The program stores a struct op * in user_data so the completion loop can find the right handler later.

io_uring_prep_recv(sqe,
                   c->fd,
                   c->req + c->req_len,
                   sizeof(c->req) - c->req_len,
                   0);
io_uring_prep_openat(sqe, AT_FDCWD, c->path, O_RDONLY, 0);
io_uring_prep_read(sqe, c->file_fd, c->body, sizeof(c->body), -1);
io_uring_prep_send(sqe,
                   c->fd,
                   c->body + c->body_sent,
                   c->body_len - c->body_sent,
                   0);
io_uring_prep_close(sqe, c->fd);

The io_uring_prep_* helpers fill the SQE with the operation the kernel should try. They do not wait for the operation. The result comes back later in cqe->res.

One Request Lifecycle

One request moves through the program in this order:

main creates the listening socket and starts workers.
accept_loop accepts conn_fd, chooses a worker, enqueues the fd, and writes to that worker’s wake_fd.
worker_main receives an OP_WAKE completion and calls register_new_connections.
register_new_connections allocates struct conn and calls submit_recv.
A receive CQE arrives, so dispatch_completion calls on_recv.
on_recv either submits another receive or calls start_response.
start_response builds headers and calls submit_open_file.
An open CQE arrives, so on_open_file stores the file descriptor or prepares a 404 response.
Header send completions advance headers_sent.
File read completions fill c->body.
Body send completions advance body_sent.
End-of-file or error calls submit_close.
The close CQE calls on_close, which frees struct conn.

With MAX_WORKERS = 1, one thread owns one ring. With MAX_WORKERS = 4, four threads own four independent rings.

Entrypoints

main starts workers and then accepts connections.

int main(void) {
    int listen_fd = listen_socket(8080);
    struct worker workers[MAX_WORKERS];
 
    start_workers(workers);
    accept_loop(listen_fd, workers);
}

The baseline uses a normal acceptor thread. io_uring can also submit accept operations; that variant comes later.

start_workers creates each worker’s ring and wakeup descriptor.

static void start_workers(struct worker workers[MAX_WORKERS]) {
    for (size_t i = 0; i < MAX_WORKERS; i++) {
        struct worker *w = &workers[i];
 
        w->wake_fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
        if (w->wake_fd < 0)
            die("eventfd");
 
        fd_queue_init(&w->new_fds);
        pthread_mutex_init(&w->mu, NULL);
 
        pthread_create(&w->thread, NULL, worker_main, w);
    }
}

worker_main creates w->ring with io_uring_queue_init. The wakeup descriptor exists before the worker starts because the acceptor may call eventfd_write as soon as it enqueues a socket. eventfd_write adds an unsigned 64-bit value to the eventfd counter. The outstanding io_uring_prep_read on wake_fd completes when the counter becomes nonzero.

static void accept_loop(int listen_fd, struct worker workers[MAX_WORKERS]) {
    size_t next = 0;
 
    for (;;) {
        int conn_fd = accept(listen_fd, NULL, NULL);
        if (conn_fd < 0) {
            if (errno == EINTR)
                continue;
            die("accept");
        }
 
        struct worker *w = &workers[next++ % MAX_WORKERS];
        enqueue_fd(w, conn_fd);
        eventfd_write(w->wake_fd, 1);
    }
}

The acceptor only assigns sockets to workers. The worker receives request bytes, parses HTTP, opens files, submits socket SQEs, and closes the connection.

Worker Completion Loop

Each worker owns one ring and one completion loop.

static void *worker_main(void *arg) {
    struct worker *w = arg;
    io_uring_queue_init(256, &w->ring, 0);
 
    submit_wake_read(w);
 
    for (;;) {
        io_uring_submit_and_wait(&w->ring, 1);
 
        struct io_uring_cqe *cqe;
        unsigned head, count = 0;
 
        io_uring_for_each_cqe(&w->ring, head, cqe) {
            struct op *op = io_uring_cqe_get_data(cqe);
            dispatch_completion(w, op, cqe);
            count++;
        }
 
        io_uring_cq_advance(&w->ring, count);
    }
}

The loop is sequential inside one worker. The difference from epoll is the event meaning:

epoll event: this fd is ready, so the application should call read or write
io_uring CQE: this operation already ran, and cqe->res is the result

Wakeups From The Acceptor

The acceptor wakes a worker by writing to wake_fd. The worker has an outstanding OP_WAKE read on the same eventfd.

static void submit_wake_read(struct worker *w) {
    struct op *op = new_op(OP_WAKE, NULL);
    struct io_uring_sqe *sqe = io_uring_get_sqe(&w->ring);
 
    io_uring_prep_read(sqe,
                       w->wake_fd,
                       &w->wake_value,
                       sizeof(w->wake_value),
                       0);
    io_uring_sqe_set_data(sqe, op);
}
 
static void on_wake(struct worker *w) {
    register_new_connections(w);
    submit_wake_read(w);
}

When the wake read completes, the worker drains the accepted-fd queue and submits the first receive for each connection.

static void register_new_connections(struct worker *w) {
    int conn_fd;
 
    while (dequeue_fd(w, &conn_fd)) {
        struct conn *c = calloc(1, sizeof(*c));
        c->fd = conn_fd;
        c->file_fd = -1;
        submit_recv(w, c);
    }
}

Dispatch

dispatch_completion is the replacement for a synchronous call stack. It looks at the operation kind and calls the corresponding on_* handler.

static void dispatch_completion(struct worker *w,
                                struct op *op,
                                struct io_uring_cqe *cqe) {
    if (op->kind == OP_WAKE) {
        if (cqe->res == sizeof(w->wake_value))
            on_wake(w);
        else
            submit_wake_read(w);
        free(op);
        return;
    }
 
    struct conn *c = op->conn;
 
    if (cqe->res < 0 &&
        op->kind != OP_OPEN_FILE &&
        op->kind != OP_CLOSE) {
        submit_close(w, c);
        free(op);
        return;
    }
 
    switch (op->kind) {
    case OP_RECV:
        on_recv(w, c, cqe->res);
        break;
    case OP_OPEN_FILE:
        on_open_file(w, c, cqe->res);
        break;
    case OP_SEND_HEADERS:
        on_send_headers(w, c, cqe->res);
        break;
    case OP_READ_FILE:
        on_read_file(w, c, cqe->res);
        break;
    case OP_SEND_BODY:
        on_send_body(w, c, cqe->res);
        break;
    case OP_CLOSE:
        on_close(w, c);
        break;
    }
 
    free(op);
}

cqe->res is either a non-negative operation result or a negative -errno value. OP_OPEN_FILE is allowed to reach on_open_file with a negative result because the handler turns “file not found” into a 404 response.

Receive Request Bytes

submit_recv asks the kernel to receive bytes into the unused part of c->req.

static void submit_recv(struct worker *w, struct conn *c) {
    struct op *op = new_op(OP_RECV, c);
    struct io_uring_sqe *sqe = io_uring_get_sqe(&w->ring);
 
    io_uring_prep_recv(sqe,
                       c->fd,
                       c->req + c->req_len,
                       sizeof(c->req) - c->req_len,
                       0);
    io_uring_sqe_set_data(sqe, op);
}

on_recv runs after the receive completes. If the request headers are incomplete, it submits another receive.

static void on_recv(struct worker *w, struct conn *c, int n) {
    if (n == 0) {
        submit_close(w, c);
        return;
    }
 
    c->req_len += (size_t)n;
 
    if (headers_complete(c)) {
        start_response(w, c);
        return;
    }
 
    submit_recv(w, c);
}

The receive loop is no longer a C while loop. It is a chain: submit receive, wait for CQE, handle completion, maybe submit another receive.

Open File And Send Headers

start_response parses the request, builds response headers, and submits an open operation for the requested file.

static void start_response(struct worker *w, struct conn *c) {
    char url_path[1024];
 
    if (parse_http_get(c->req, c->req_len, url_path, sizeof(url_path)) < 0) {
        c->headers_len = build_404(c->headers, sizeof(c->headers));
        c->close_after_headers = 1;
        submit_send_headers(w, c);
        return;
    }
 
    snprintf(c->path, sizeof(c->path), "%s%s", DOCROOT, url_path);
    c->headers_len = build_ok_headers(c->headers,
                                      sizeof(c->headers),
                                      mime_for(url_path));
    submit_open_file(w, c);
}

submit_open_file asks the kernel to open the file.

static void submit_open_file(struct worker *w, struct conn *c) {
    struct op *op = new_op(OP_OPEN_FILE, c);
    struct io_uring_sqe *sqe = io_uring_get_sqe(&w->ring);
 
    io_uring_prep_openat(sqe, AT_FDCWD, c->path, O_RDONLY, 0);
    io_uring_sqe_set_data(sqe, op);
}

on_open_file either stores the file descriptor or changes the response to 404.

static void on_open_file(struct worker *w, struct conn *c, int fd) {
    if (fd < 0) {
        c->headers_len = build_404(c->headers, sizeof(c->headers));
        c->close_after_headers = 1;
        submit_send_headers(w, c);
        return;
    }
 
    c->file_fd = fd;
    submit_send_headers(w, c);
}

Header sends can complete partially, so the connection tracks headers_sent.

static void submit_send_headers(struct worker *w, struct conn *c) {
    struct op *op = new_op(OP_SEND_HEADERS, c);
    struct io_uring_sqe *sqe = io_uring_get_sqe(&w->ring);
 
    io_uring_prep_send(sqe,
                       c->fd,
                       c->headers + c->headers_sent,
                       c->headers_len - c->headers_sent,
                       0);
    io_uring_sqe_set_data(sqe, op);
}
 
static void on_send_headers(struct worker *w, struct conn *c, int n) {
    if (n == 0) {
        submit_close(w, c);
        return;
    }
 
    c->headers_sent += (size_t)n;
 
    if (c->headers_sent < c->headers_len) {
        submit_send_headers(w, c);
        return;
    }
 
    if (c->close_after_headers) {
        submit_close(w, c);
        return;
    }
 
    submit_file_read(w, c);
}

Read File And Send Body

submit_file_read asks the kernel to read a file chunk into c->body. The worker does not call read(c->file_fd, c->body, sizeof(c->body)) directly.

static void submit_file_read(struct worker *w, struct conn *c) {
    struct op *op = new_op(OP_READ_FILE, c);
    struct io_uring_sqe *sqe = io_uring_get_sqe(&w->ring);
 
    io_uring_prep_read(sqe,
                       c->file_fd,
                       c->body,
                       sizeof(c->body),
                       -1);
    io_uring_sqe_set_data(sqe, op);
}

on_read_file sends the chunk, or closes when the file reaches end-of-file.

static void on_read_file(struct worker *w, struct conn *c, int n) {
    if (n == 0) {
        submit_close(w, c);
        return;
    }
 
    c->body_len = (size_t)n;
    c->body_sent = 0;
    submit_send_body(w, c);
}

Body sends also complete partially.

static void submit_send_body(struct worker *w, struct conn *c) {
    struct op *op = new_op(OP_SEND_BODY, c);
    struct io_uring_sqe *sqe = io_uring_get_sqe(&w->ring);
 
    io_uring_prep_send(sqe,
                       c->fd,
                       c->body + c->body_sent,
                       c->body_len - c->body_sent,
                       0);
    io_uring_sqe_set_data(sqe, op);
}
 
static void on_send_body(struct worker *w, struct conn *c, int n) {
    if (n == 0) {
        submit_close(w, c);
        return;
    }
 
    c->body_sent += (size_t)n;
 
    if (c->body_sent < c->body_len) {
        submit_send_body(w, c);
        return;
    }
 
    submit_file_read(w, c);
}

After each body send, the worker submits the next file read and returns to the completion loop. Other completions can be processed while file I/O is outstanding.

Close

The worker frees struct conn only after the socket close completion arrives. The example closes file_fd directly before submitting the socket close; a stricter version can submit a separate close operation for the file descriptor too.

static void submit_close(struct worker *w, struct conn *c) {
    struct op *op = new_op(OP_CLOSE, c);
    struct io_uring_sqe *sqe = io_uring_get_sqe(&w->ring);
 
    if (c->file_fd >= 0) {
        close(c->file_fd);
        c->file_fd = -1;
    }
 
    io_uring_prep_close(sqe, c->fd);
    io_uring_sqe_set_data(sqe, op);
}
 
static void on_close(struct worker *w, struct conn *c) {
    (void)w;
    free(c);
}

Shared Ring Variant

Multiple threads can coordinate around one ring, but the ownership protocol must be explicit.

If two threads consume completions from the same completion queue and both can resume the same struct conn, the application needs synchronization around that connection. A per-worker-ring design avoids that because completions for a connection arrive on the ring owned by that connection’s worker.

Shared rings can make sense for specialized designs, but they require a deliberate ownership protocol.

Multishot Accept Variant

The baseline architecture used a normal acceptor thread. io_uring can also submit an accept operation. With multishot accept, one submitted accept can produce multiple CQEs as new connections arrive.

Multishot accept changes only the accept path. Once a connection is assigned to a worker, the same submit_* / on_* chain owns the request lifecycle.

Sources

Linux man-pages: io_uring(7), io_uring_setup(2), and io_uring_prep_multishot_accept(3).

Edmondo's Vault

Explorer

io_uring Server Architecture

Full Program Skeleton

State And Ownership

Core io_uring Calls

One Request Lifecycle

Entrypoints

Worker Completion Loop

Wakeups From The Acceptor

Dispatch

Receive Request Bytes

Open File And Send Headers

Read File And Send Body

Close

Shared Ring Variant

Multishot Accept Variant

See also

Sources

Graph View

Table of Contents

Backlinks