File System Monitoring

File system monitoring refers to the process of tracking changes to files and directories in real-time. This capability is essential for applications that need to respond to modifications in the file system, such as build systems, logging frameworks, or security software. In Linux, monitoring is primarily supported by inotify, fanotify, and their integration with epoll for efficient event handling.

File System Events

File system monitoring is concerned with various types of events. These include file access, modification, creation, deletion, and metadata updates. In addition to files, changes to directories—such as adding, renaming, or removing entries—are also crucial for many use cases.

The idea is to have a dedicated thread (or part of your application) that uses epoll, poll, or select to monitor the inotify file descriptor or another monitoring mechanism. This thread is blocked until the file descriptor becomes ready for reading, which happens whenever the kernel queues new events for the monitored files or directories.

Mechanisms for File System Monitoring

inotify: Fine-Grained Monitoring

inotify is a Linux kernel API designed to monitor file and directory events. When an application initializes an inotify instance, it receives a file descriptor that acts as an event queue. Watches can be added to specific files or directories to track selected event types. To know when it’s time to read from the inotify file descriptor, you use I/O multiplexing mechanisms, such as epoll, poll, or select.

fanotify: File System-Level Monitoring

Introduced in later Linux kernels, fanotify operates at a higher abstraction level than inotify. Instead of tracking specific files or directories, fanotify can monitor entire file systems or mounts. This scalability makes it particularly suited for applications like antivirus scanners or file system auditing.

fanotify provides access control capabilities, allowing applications to block file access until a policy decision is made. However, it provides less detailed event information than inotify and may be overkill for small-scale monitoring tasks.

Tip

fanotify can handle million of events across large filesystems while inotify struggles when monitoring hundreds of thousands of files due to the memory overhead of each watch

Tip

Since fanotify can intercept file access request and allow/deny them depending on application-defined policies, it is ideal for antivirus scanners or file-access auditing

Resource limits

In/proc/sys/fs there are limits that affect inotify and, to some extent, fanotify:

• max_user_instances: This limits the maximum number of inotify instances (i.e., inotify_init() calls) that a single user can create. Each instance corresponds to a unique event queue. This helps prevent users from creating excessive instances that could exhaust kernel memory.
• max_user_watches: This parameter controls the maximum number of watches (monitored files or directories) a user can create across all inotify instances. Each watch consumes kernel memory, so this limit protects the system from resource exhaustion, especially when monitoring large directory trees.
• max_queued_events: This sets the maximum number of events that can be queued for a single inotify instance. When the queue is full, further events are dropped until space is available. This parameter ensures that a flood of file system events doesn’t overwhelm the kernel or the application.

You can check and adjust these values dynamically via the /proc/sys/fs/ interface. For example:

cat /proc/sys/fs/inotify/max_user_instances
cat /proc/sys/fs/inotify/max_user_watches
cat /proc/sys/fs/inotify/max_queued_events

To modify them:

sudo sysctl fs.inotify.max_user_watches=524288

Integration with epoll

Both inotify and fanotify produce events via file descriptors, which can be monitored using epoll. This integration is essential for building efficient, event-driven applications. epoll allows multiplexing multiple descriptors, avoiding the inefficiency of polling each descriptor individually.

For example, a single application might monitor both network sockets and file system events using epoll, enabling responsive and resource-efficient designs.

Example with inotify

When you initialize an inotify instance using inotify_init, it returns a file descriptor (fd) that represents the event queue.

int fd = inotify_init();

You then add watches for specific files or directories using inotify_add_watch.

int wd = inotify_add_watch(fd, "/path/to/watch", IN_MODIFY | IN_CREATE | IN_DELETE);

Lastly, you use a system call like epoll, poll, or select to monitor the fd for readiness. These mechanisms notify you when there is data to read (i.e., when events have occurred). Example using ** epoll** (recommended for scalability):

struct epoll_event event, events[10];
int epfd = epoll_create(1);
 
event.events = EPOLLIN;  // Ready for reading
event.data.fd = fd;
epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &event);
 
// Wait for events
int n = epoll_wait(epfd, events, 10, -1);
for (int i = 0; i < n; i++) {
    if (events[i].data.fd == fd) {
   	 // Read the inotify events
    }
}

Example using poll:

struct pollfd pfd;
pfd.fd = fd;
pfd.events = POLLIN;
 
int ret = poll(&pfd, 1, -1);  // Wait indefinitely
if (pfd.revents & POLLIN) {
    // Read the inotify events
}

Example using select (less efficient for many descriptors):

fd_set fds;
FD_ZERO(&fds);
FD_SET(fd, &fds);
 
select(fd + 1, &fds, NULL, NULL, NULL);  // Wait indefinitely
if (FD_ISSET(fd, &fds)) {
    // Read the inotify events
}

Once notified, read from the fd using the read system call. This retrieves a buffer of inotify_event structures representing the file system changes.

char buffer[1024];
ssize_t len = read(fd, buffer, sizeof(buffer));
 
struct inotify_event *event = (struct inotify_event *)buffer;
while (len > 0) {
    // Process the event
    printf("Event on %s: %d\n", event->name, event->mask);
    len -= sizeof(struct inotify_event) + event->len;
    event = (struct inotify_event *)((char *)event + sizeof(struct inotify_event) + event->len);
}