Edmondo's Vault

Linux Networking

Aug 16, 202615 min read

Linux networking is Linux’s implementation of an Internet host network stack. This note assumes Ethernet, IP prefix notation, TCP, and UDP fundamentals. Its subject is the Linux-specific mapping from those protocols to kernel state: the ip command changes networking configuration, ss inspects sockets, and applications use socket system calls to exchange data.

Reference models and the Linux implementation

ISO/IEC 7498-1 defines the seven-layer Open Systems Interconnection (OSI) Basic Reference Model. It provides a vocabulary for locating networking functions rather than an implementation specification. Internet hosts follow the Internet protocol suite more directly: RFC 1122 groups communication into link, Internet, and transport layers and warns that real implementations need not preserve strict internal boundaries between them.

Linux implements Ethernet, IP, TCP, UDP, and other protocols; it does not contain seven kernel modules corresponding to the seven OSI layers. The Internet suite supplies the protocols, while Linux chooses the internal objects and administration APIs:

OSI reference areaInternet-suite areaLinux representationPrimary userspace interface
Physical (L1)Link-specific hardwareDriver and physical link stateethtool for driver and hardware settings
Data link (L2)LinkNetwork device, media access control (MAC) address, and maximum transmission unit (MTU)ip link
Network (L3)InternetAddresses attached to devices, IP-to-link-layer neighbour records, and routesip address, ip neighbour, ip route
Transport (L4)TransportTCP and UDP socketssocket system calls, ss

Linux network state

A network namespace is one instance of the Linux network stack. At boot, the kernel creates one initial network namespace. PID 1 starts in it, and a new process normally inherits its parent’s network namespace, so a system that has created no others has exactly one.

Programs can create more network namespaces with clone(CLONE_NEWNET) or unshare(CLONE_NEWNET), and a thread can enter an existing one with setns. Containers are a common use of this mechanism, not its definition: a container runtime combines a network namespace with other isolation and resource-control features. Kubernetes normally creates one network namespace per Pod; every container in that Pod shares its interfaces, addresses, routes, and port space. A Pod using host networking remains in the node’s initial network namespace.

Each network namespace has its own network devices, interface addresses, neighbour tables, routes, firewall state, sockets, and TCP or UDP port space. Every network device belongs to one network namespace at a time. A socket belongs to the namespace in which the process created it and remains there if the process later changes namespace. Linux Namespaces explains the general creation, membership, and lifetime mechanism shared by all Linux namespace types.

The number of network namespaces is therefore dynamic rather than a fixed container count. These commands answer three different inspection questions:

# Which network namespace contains this shell?
readlink /proc/self/ns/net
# net:[4026531840]
 
# Which network namespaces are visible through processes in this /proc view?
lsns --type net
 
# Which network namespaces have names registered under /run/netns?
ip netns list

ip netns list can be empty even when unnamed namespaces created by container runtimes or other programs exist. Within one network namespace, Linux connects networking records through identifiers stored in those records:

ObjectIdentity or lookup keyRelation to other objects
Network deviceInterface index plus a userspace name such as service0Address records and routes refer to its index
Interface addressAddress family, prefix, and interface indexAttached to one network device; may cause the kernel to create prefix and local routes
RouteTable, destination prefix, type, priority, and other selectorsMay name an output-interface index and next hop; carries a tag identifying who installed it
SocketProtocol, local and remote addresses and portsBelongs to one network namespace; may bind to an address or a device

Network devices

A network device is the kernel object represented internally by struct net_device. Both a physical network interface card (NIC) and a virtual interface created by Linux use this base object. The object stores an interface index, userspace-visible name, maximum transmission unit (MTU), link-layer properties, and an administrative UP or DOWN state that says whether Linux may use it. Linux Virtual Network Interfaces defines the available virtual device types and what each endpoint connects to.

A Linux bridge is one virtual network-device type. It forwards Ethernet frames between member ports by consulting a MAC-to-port table. Because the bridge is also a Linux network device, the kernel can store interface-address records on it. Those addresses are used by the host IP stack; they do not change the bridge’s MAC-based frame forwarding into IP routing. Linux Virtual Network Interfaces owns the full bridge mechanism.

The interface index identifies the device to the kernel for the lifetime of that device. Names such as eno1 and service0 are labels that userspace tools resolve to an index. These objects appear under /sys/class/net; the dev keyword in the ip command is unrelated to the /dev directory.

Interface addresses

The kernel keeps interface-address records on each network device. After ip address add 192.0.2.10/32 dev service0, the device identified by service0 has a record containing the IPv4 address 192.0.2.10, prefix length 32, and the device’s interface index. Optional fields control properties such as where the address is valid and how long it remains usable. Deleting the device deletes its attached address records.

Attaching the record to one device does not require every packet for that address to arrive through that device. Ingress validation separately decides whether a destination must belong to the incoming interface (the strong-host model) or may belong to any interface on the host (the weak-host model).

The interface relation is required even for a host-only /32. Linux uses it to decide:

  • which device owns the address and loses it when that device is deleted;
  • which interface-specific IPv4 or IPv6 rules and address options apply;
  • which device a derived connected-prefix or local route refers to;
  • which interface-specific address records are returned by inspection and notifications.

The kernel cannot infer this relation from the numeric address. A host can have several candidate devices, no route may exist yet, and adding the address can itself create routing entries. An address that must survive changes to physical links still needs a virtual network device with the required lifetime; this example uses the dummy interface named service0, which has no external peer.

Routes

A route is a record the kernel consults to decide what to do with a packet whose destination matches the route’s prefix. Linux first uses routing-policy rules to choose a routing table, then looks for a matching route in that table. The record can say to deliver the packet locally, reject it, or send it through an output device and optional next hop. It also stores fields such as priority, preferred source address, and provenance identifying who installed the route.

Assigning 192.0.2.10/32 to service0 creates a route in the built-in local table whose action is to deliver packets for that address inside this host. The later section on ip route show maps its printed fields to the stored route record.

Sockets

socket, bind, connect, send, and recv create or operate on socket state through file descriptors. ss reads Linux socket-diagnostic interfaces to inspect protocols, addresses, ports, queues, and connection state. The TCP Stack owns TCP protocol behaviour rather than this Linux administration interface.

Interacting with Linux network state

iproute2 is the userspace tool suite that provides ip and ss. The ip program does not have a dedicated system call for each network object. It opens a socket and exchanges messages with the kernel.

The C socket API is socket(domain, type, protocol). Its three arguments select different parts of the communication channel:

int fd = socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE);

The call chooses the channel in this order:

socket parameterValueSelection
Address familyAF_NETLINKUse Linux’s local kernel/userspace messaging interface rather than an IPv4 or IPv6 socket
Socket typeSOCK_RAWOpen a raw socket; Netlink currently treats it and the datagram type SOCK_DGRAM equivalently, and userspace handles the Netlink headers
Netlink protocolNETLINK_ROUTESend messages to the kernel subsystem for network devices, addresses, neighbours, routes, and related state

AF_NETLINK therefore selects Netlink, the general socket-based mechanism. NETLINK_ROUTE selects routing Netlink (rtnetlink) within that mechanism. Despite its name, NETLINK_ROUTE handles link and address state as well as routes. These messages remain inside the host; no Ethernet or IP packet is transmitted through a network device.

The two constants occupy different positions in the API. Keeping the address family fixed while changing the third argument opens a socket connected to a different kernel Netlink protocol:

int network_state = socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE);
int audit_events  = socket(AF_NETLINK, SOCK_RAW, NETLINK_AUDIT);

Both descriptors use Netlink message framing and addressing. Messages sent on network_state go to the rtnetlink handlers; messages sent on audit_events go to the Linux audit subsystem. The RTM_* operation is selected only after the rtnetlink socket exists.

After opening the socket, the nlmsg_type field in each message header selects an operation. This request asks rtnetlink to return every link record:

#include <linux/netlink.h>
#include <linux/rtnetlink.h>
#include <sys/socket.h>
 
struct {
    struct nlmsghdr header;
    struct ifinfomsg link;
} request = {
    .header = {
        .nlmsg_len = NLMSG_LENGTH(sizeof(struct ifinfomsg)),
        .nlmsg_type = RTM_GETLINK,
        .nlmsg_flags = NLM_F_REQUEST | NLM_F_DUMP,
        .nlmsg_seq = 1,
    },
    .link = {
        .ifi_family = AF_UNSPEC,
    },
};
 
struct sockaddr_nl kernel = { .nl_family = AF_NETLINK };
sendto(network_state, &request, request.header.nlmsg_len, 0,
       (struct sockaddr *)&kernel, sizeof(kernel));

NLM_F_REQUEST marks a userspace request. NLM_F_DUMP asks for all matching objects rather than one record. The zero-initialized nl_pid in kernel addresses Netlink port ID 0, which denotes the kernel. A production caller must check socket and sendto, receive all multipart replies, match nlmsg_seq, and handle NLMSG_ERROR and NLMSG_DONE.

The following abridged syscall trace shows the narrower request made by ip link show dev lo on the current host:

socket(AF_NETLINK, SOCK_RAW|SOCK_CLOEXEC, NETLINK_ROUTE) = 4
sendmsg(4, {nlmsg_type=RTM_GETLINK,
            ifi_family=AF_UNSPEC, IFLA_IFNAME="lo"}) = 48
recvmsg(4, {nlmsg_type=RTM_NEWLINK,
            ifi_index=if_nametoindex("lo"), IFLA_IFNAME="lo"}) = 1064

SOCK_CLOEXEC asks Linux to close the file descriptor if the process later executes another program; it does not select a Netlink subsystem. AF_UNSPEC means that the link query is not restricted to IPv4 or IPv6. strace renders the returned numeric interface index as if_nametoindex("lo") to make the value readable.

The first trace line opens the rtnetlink socket. The second sends a read request to the kernel, whose Netlink port ID is 0. Its common nlmsghdr header contains RTM_GETLINK; the following ifinfomsg record identifies this as a link request, and the IFLA_IFNAME attribute selects lo. The third line receives the current link record. rtnetlink uses RTM_NEWLINK both for a request that creates or changes a link and for a reply that describes an existing link; this reply does not mutate lo.

Every rtnetlink message uses the same three-part shape shown by the trace:

Message partLink-query valuePurpose
Common Netlink headernlmsghdr with RTM_GETLINKLength, operation, flags, sequence number, and sender port ID
Object-specific recordifinfomsgFixed fields shared by link operations
Typed attributesIFLA_IFNAME="lo"Variable fields, each encoded with a type, length, and value

The object determines the GET request and the NEW message used to create, change, or return its records. Deletion uses the corresponding DEL message:

ip objectRead requestCreate, change, or returned record
ip linkRTM_GETLINKRTM_NEWLINK
ip addressRTM_GETADDRRTM_NEWADDR
ip neighbourRTM_GETNEIGHRTM_NEWNEIGH
ip routeRTM_GETROUTERTM_NEWROUTE

The single frontend spans L2 and L3 because rtnetlink exposes both sets of Linux objects. For a physical Ethernet port, ip link set dev eno1 up changes the administrative state of its network-device object, and the driver applies that state to the hardware. ethtool uses a separate Linux API for settings owned by the driver or hardware, such as link speed and hardware-assisted packet processing. These command boundaries follow Linux API ownership rather than OSI layers.

Address assignment

Every ip address add operation attaches an address record to a network device. There is no form of this operation that adds an address to the host without selecting a device, so dev does not choose among different kinds of address owner.

For an address operation, the object-specific part of the rtnetlink message is an ifaddrmsg record. It contains the address family (IPv4 or IPv6), prefix length, scope (where the address is valid), flags, and ifa_index, which is the numeric interface index of the target device. Fields following that record carry the address itself and optional values such as a peer address, broadcast address, and usable lifetime.

The address command maps its arguments onto an RTM_NEWADDR request:

ip address add <address>/<prefix-length> dev <interface-name>
Command partRequest meaning
ip address addConstruct an RTM_NEWADDR request
192.0.2.10/32Supply the local address and ifa_prefixlen
dev service0Resolve service0 to its interface index and supply ifa_index

The kernel request format requires the interface index because the operation creates an address record on one particular network device. The literal word dev adds no further semantic choice: it is an iproute2 parser marker for the following interface name. iproute2 resolves that name and sends the numeric ifa_index; the kernel never receives the word dev.

iproute2 could have defined the mandatory interface as an unlabelled positional argument, for example ip address add ADDRESS IFNAME. Its actual grammar labels fields with keywords such as dev, peer, broadcast, scope, label, and valid_lft, so the parser requires dev IFNAME. The explicit keyword makes the grammar regular among many optional clauses; it does not reveal a second possible target for the address.

The same dev keyword labels different interface fields elsewhere in iproute2:

ContextMeaning of dev service0
ip link set dev service0 upSelect the network-device object to mutate
ip address add ADDRESS dev service0Supply the owning interface index for the address record
ip route add PREFIX dev service0Supply the route’s output-interface index
ip route show outputPrint the interface stored in the route record
ip route get outputPrint the output device returned by a resolved lookup

Stored routes and resolved lookups

ip route show asks the kernel to return stored routing-table records. The address on service0 caused the kernel to create this record:

ip route show table local 192.0.2.10
# local 192.0.2.10 dev service0 proto kernel scope host src 192.0.2.10

The output prints the corresponding route fields:

FieldLinux route field and effect
local 192.0.2.10rtm_type = RTN_LOCAL: deliver this destination inside the host
dev service0Output-interface index associated with the device that owns the address
proto kernelrtm_protocol = RTPROT_KERNEL (2): the kernel installed the route
scope hostrtm_scope = RT_SCOPE_HOST: the destination is reachable inside this host
src 192.0.2.10Preferred source address for matching locally originated traffic

rtm_protocol stores route provenance, although ip route prints the shorter label proto. Here the kernel’s IPv4 address code installed the local route while processing the address addition and tagged it RTPROT_KERNEL. ip route show proto kernel filters on that value. Other values identify routes added by an administrator, learned by a routing program, or created from an IPv6 Router Advertisement. Every one of these routes still lives in a kernel routing table; the field records its origin rather than the packet protocol.

ip route get sends RTM_GETROUTE and asks the kernel to resolve a hypothetical packet. Without an input-interface argument it models locally originated traffic:

ip route get 192.0.2.10
# local 192.0.2.10 dev lo src 192.0.2.10

The destination is an address of this host, so Linux does not transmit the packet through service0. The result reports dev lo: lo is Linux’s loopback network device, used here to represent delivery from the host back to the same host. This does not move the configured address to lo; the address record and the stored local route still refer to service0. A packet received from another machine still records the device on which it arrived before Linux delivers it to a local socket.

Persistent configuration

ip changes live kernel state. A userspace manager such as NetworkManager stores a higher-level connection profile and issues Netlink requests to recreate devices, addresses, and routes after boot. It coordinates Domain Name System (DNS) resolver configuration through separate resolver interfaces. NetworkManager is Linux administration software rather than part of the OSI or Internet protocol stack.

Two managers configuring the same object can replace each other’s addresses or routes. A deployment must therefore choose which manager recreates each device, address, and route after boot.

Sources

See also

Graph View

Backlinks