Linux Devices and Drivers: Finally Explained Simply

The Linux kernel talks to hardware through a layered system that can be hard to see from user space. When you plug in a USB drive or a printer, a chain of code runs in the background to make it work. That chain is what the kernel calls the device model (the kernel’s internal representation of every piece of hardware attached to the system), and understanding it changes how you debug hardware issues on Linux.

I spent a long time confused by the term “driver.” I thought a driver was a single piece of code that talked to a specific piece of hardware. That mental model is wrong, or at least incomplete. In Linux, hardware support is split across several layers, and the driver is just one of them. This guide explains the layers, the terminology, and the tools you use to inspect them.

The four layers

A piece of hardware in Linux is supported by a stack of components. From the top down, they are:

• The character or block device interface. This is the file in /dev/ that user programs open.
• The kernel subsystem the device belongs to. Input, network, storage, sound, and so on.
• The bus driver, which handles the protocol for talking to devices on a particular bus (a communication pathway between the CPU and peripherals, like USB, PCI, or I2C).
• The device-specific driver, which knows the registers and commands for one particular chip.

When you read a file in /dev/sda1, the request flows down through these layers. The filesystem asks the block layer (the kernel subsystem that handles block devices like disks) for a block. The block layer asks the disk driver. The disk driver asks the SATA (Serial ATA, a common disk interface) host controller. The host controller talks to the disk over the SATA bus.

Each layer is independent. The same disk driver can sit behind a SATA controller, a USB controller, or a RAID (redundant array of inexpensive disks, a way to combine multiple disks for performance or reliability) controller. The same SATA controller can have many different disks attached. The same block device interface can be served by spinning disks, SSDs (solid-state drives), network block devices, or RAM disks. The layers are what make this composition possible.

The device tree: describing hardware

On embedded systems (small, purpose-built computers like routers, single-board computers, and IoT devices) and on systems with a device tree, the hardware description lives in a file called the device tree. The device tree is a structured data file that tells the kernel what hardware exists and how it is wired up. The kernel reads the device tree at boot and uses it to figure out which drivers to load.

The device tree is necessary because embedded systems have hardware configurations that cannot be detected at runtime. A Raspberry Pi has a known set of GPIO (general-purpose input/output) pins, a known I2C bus layout, and a known set of peripherals. The device tree encodes that knowledge. The kernel uses the device tree to match hardware to drivers.

For developers, the device tree is a place to look when hardware is not working. If a peripheral is not detected, the device tree might not describe it, or the description might be wrong. Tools like fdtdump can decode the compiled device tree blob (/boot/dtb/*.dtb) into a human-readable form. Comparing the actual device tree to the expected one is a common debugging step.

For desktop and server systems, the device tree is less common. ACPI (Advanced Configuration and Power Interface, a standard for hardware discovery and power management on x86 systems) is the equivalent. ACPI tables describe the hardware layout and the kernel reads them at boot. The user-facing tools are different, but the underlying idea is the same: a description of what hardware exists.

Loading and managing drivers

Drivers in Linux are loadable kernel modules. A kernel module (a piece of code that can be loaded into the kernel at runtime to add functionality) is a binary file that extends the kernel with new functionality. Most distributions ship with thousands of modules in /lib/modules/$(uname -r)/. The kernel loads a module when it detects hardware that needs it, and unloads it when the hardware is gone.

The lsmod command lists currently loaded modules. The output looks like this:

Module                  Size  Used by
nvidia              12345678  0
snd_hda_intel         123456  3
snd_hda_codec        234567  1 snd_hda_intel

The Used by column shows which other modules depend on this one. In the example, the audio codec module is used by the Intel HDA driver, which is itself used by nothing (yet).

To load a module manually, use modprobe <name>. modprobe understands dependencies and loads any modules the requested one needs. To unload, use modprobe -r <name>. To see what a module supports, check /sys/module/<name>/. That directory has parameters, references, and statistics.

To find which driver is bound to a specific piece of hardware, use lspci -k (for PCI devices, the Peripheral Component Interconnect bus used for internal cards), lsusb -v (for USB devices), or lshw (for everything). The -k flag to lspci shows the kernel driver in use.

The kernel log: dmesg

The kernel ring buffer (a fixed-size memory area where the kernel stores its most recent log messages) is the place to look when something hardware-related goes wrong. dmesg prints the buffer. The most recent messages are at the bottom. To follow new messages, use dmesg -w.

The kernel ring buffer is finite. On a busy system, old messages get overwritten. For long-term logging, configure your syslog to capture kernel messages. The rsyslog and journald configurations have separate handling for kernel messages.

Common things to look for in dmesg:

• “driver X bound to device Y”: a driver successfully attached to a device
• “probe of Y failed with error -Z”: a driver tried to initialize a device and failed
• “disabling IRQ X”: the kernel gave up on a hardware interrupt (a signal the hardware uses to get the CPU’s attention)
• “I/O error, dev sda, sector N”: a disk operation failed

Each of these tells a different story. The first is good news. The second usually means a hardware or firmware problem. The third is rare and usually indicates flaky hardware. The fourth is a disk problem and warrants immediate backup.

Driver parameters and sysfs

Many kernel modules accept parameters. The parameters control behavior at load time. For example, the nvidia module accepts NVreg_OpenRmEnableUnsupportedGpus to allow the driver to load on unsupported hardware. The snd_hda_intel module accepts model to force a specific audio codec configuration.

You set module parameters at load time, either in /etc/modprobe.d/ (a directory of configuration files that modprobe reads) or on the kernel command line. The /etc/modprobe.d/ directory has files like mymodule.conf with contents like options mymodule param=value.

Once the module is loaded, some parameters are exposed through sysfs (a virtual filesystem at /sys/ that exposes kernel objects and their attributes as files and directories), the virtual filesystem mounted at /sys/. The directory /sys/module/<name>/parameters/ contains one file per parameter. You can read the current value and, in some cases, write a new value to change the behavior at runtime.

This is the Linux way. The kernel exposes its state as files. You can read those files, write to them, and use shell tools to inspect and modify the running system. The principle applies to device drivers just as it does to network interfaces and CPU frequency scaling.

The new drivers: a guide for contributors

Writing a Linux driver is a project in itself, but the basic shape is approachable. The Linux kernel has a documented driver model, a set of helper functions, and a coding style. A simple character driver (a driver that exposes a device as a stream of bytes you can read and write) can be a few hundred lines of C.

The kernel’s documentation in Documentation/driver-api/ is the starting point. It covers the device model, the basic infrastructure, and the patterns for different kinds of drivers. The kernel newbies project and the Linux driver tutorial by Bootlin are good external resources.

For most developers, the right approach is to start by understanding an existing driver for similar hardware, then adapt it. The kernel source is well-commented, and most subsystems have maintainers who review new drivers and answer questions on the mailing list.

Trade-offs

The Linux device model is more complex than a “driver per device” mental model would suggest, and the complexity has costs:

• A bug at any layer can manifest as a problem in user space, far from the root cause. Debugging requires understanding the layers.
• Driver parameters and module options are scattered across many files. Finding the right knob to turn takes research.
• The kernel ring buffer is small. Long-term hardware issues need to be captured elsewhere.
• Hardware vendors often ship closed-source drivers that bypass the standard model. They work, but they are harder to inspect and debug.

The upside is that the model is consistent. The same tools, the same files, the same patterns apply across most of the hardware in a typical system. Once you have learned the layers, you can reason about hardware you have never seen before.

When to dig into the device model

Dig in when you are debugging a hardware issue, when you are writing a driver, or when you are curious about how a system works. Skip the deep background if you are a user-space developer whose code never touches the kernel directly. The device model is interesting, but it is not something you need to know to be productive as an application developer.

For the rest of us, the device model is the layer where Linux meets the physical world. Understanding it makes the rest of the system more legible.