Edmondo's Vault

Tokio Runtime

May 18, 20264 min read

In Rust’s asynchronous programming model, responsibilities are divided between the standard library and asynchronous runtimes like Tokio. Here’s a breakdown of these responsibilities: The standard library defines Future and Waker traits, but the runtime is responsible for:

  • task scheduling
  • asynchronous io
  • timer facilities
  • synchronization primitives such as mutexes and channels
  • waking function implementation

History

Initial version 0.1

The initial version of the Tokio Scheduler had three key weaknesses:

  • it assumed that processor threads should be shut down if idle for sometime
  • it had a single thread colocated with I/O selector and a separate thread-pool for CPU bound operations that would shut down threads when idle
  • it used multiple-queues based on Chase-Lev deque, with one deque assigned to each worker thread, to support work stealing (using the crate crossbeam)

Tip

Since the IO selector was always active, it made sense to save resources shutting down threads that were used for CPU bound tasks if there were no tasks

The New scheduler

The new scheduler has several improvements:

  1. it uses a new task system in the standard library which benefits from a smaller Waker implementation and the usage of a custom vtable which uses function pointers.
  2. It provides a better queue than the Chase-Lev deque used in 0.1, inspired by the GoLang scheduler. Such a queue is managed by a single thread, and overflows into a global queue
  3. It provides an optimization for message-passing, where if a message is sent by a task and this message makes another task become runnable (because the channel is the resource that is blocking the other task, and sending a message unblock it), then that task is kept in a special slot and scheduled immediately rather than pushed to the end of runnable tasks (note: if there is already a task in this slot, that task is moved to the end)
  4. It throttles stealing by limiting the number of concurrent processors performing steal operation, using an atomic integers that is bounded
  5. To additionally reduce the synchronization between threads, while the old scheduler would notify processors as soon as a new task was added to the queue, the new one do that only if there are no processors in searching state
  6. It reduces allocation by specializing the Task in the type of Future Task<T> rather than wrapping a Box<dyn Future>. It also put hot data before the future and cold data after. When the CPU deferences the task pointer, it will load a cache line sized amount of data, so putting the header before maximize the probavbility that the header is fetched
  7. It provides an API to wake and consume the Waker, reducing the number of copies.

Spawning and Coordination in Tokio

Tokio provides a structured way to manage asynchronous tasks, ensuring that computational workloads and I/O-bound operations are efficiently scheduled. The key API for creating concurrent tasks is tokio::spawn, which launches a lightweight task on Tokio’s executor. Unlike OS threads, spawned tasks are cooperatively scheduled, meaning they yield when awaiting I/O or other blocking operations. This ensures that the runtime does not block on a single task, improving overall responsiveness.

tokio::spawn(async {
    let data = fetch_data().await;
    process_data(data);
});

In this example, the function fetch_data() runs asynchronously, and the Tokio runtime schedules other tasks while waiting for it to complete.

A critical challenge in concurrent execution is task coordination. Tokio provides tokio::join!, which allows multiple tasks to be awaited in parallel, but unlike spawn, it runs them in the same execution context, avoiding overhead from scheduling. This is useful when multiple computations must complete before proceeding.

let (result1, result2) = tokio::join!(async_task1(), async_task2());

This ensures that both tasks run concurrently, but neither is detached from the execution context.

In contrast, tokio::select! allows a task to wait on multiple futures, responding to the first one that completes. This makes it essential for handling timeouts, shutdown signals, and racing multiple I/O operations efficiently.

tokio::select! {
    _ = async_task() => println!("Task completed"),
    _ = tokio::time::sleep(Duration::from_secs(5)) => println!("Timeout!"),
}

Here, either async_task() completes first, or the timeout triggers after 5 seconds, whichever happens first.

The task system integrates closely with Tokio’s reactor and I/O selector, which efficiently multiplex events. The I/O model in Tokio is based on epoll/kqueue/io_uring, allowing tasks to be woken only when necessary, preventing excessive polling. This architecture ensures that applications can handle high concurrency while minimizing CPU overhead.

Graph View

Backlinks