Speed Up Your Linux RAID Resyncs

I rebuilt a 6TB mirror last weekend after a routine scrub flagged a sector, and the default resync speed took 14 hours. I then rebuilt the same array with the four tweaks below and it finished in 3 hours. The array did not change. The disks did not change. The only thing that changed was how much of mdadm’s default safety throttle I was willing to push.

This article walks through the four techniques that actually matter, in the order I would apply them. Each one is reversible. Each one has a downside. And none of them require anything beyond a root shell, a working mdadm, and a willingness to read the man page once.

Why resync speed matters in the first place

Every minute your array is rebuilding is a minute it is more vulnerable to a second disk failure. On a 4TB mirror, a default resync can take the better part of a day. That is a long time to live without redundancy. Faster resyncs also matter when you are doing routine scrubs (a periodic read of every block on every disk to catch silent corruption early), because the longer a scrub runs, the more it competes with normal disk I/O and the more wear it puts on your drives. The trade-off is that pushing resync speed too aggressively can saturate your disks and slow down whatever else the server is doing, so the goal is to find a sensible middle ground rather than max out every setting.

There is a second reason this matters: most home servers run their arrays in degraded mode for hours at a time without the owner knowing. A scheduled scrub runs once a week, the rebuild is invisible to anyone not logged in, and the array sits vulnerable to a second failure until the rebuild finishes. Cutting that window from 14 hours to 3 is the difference between “a second disk failure would have been recoverable” and “a second disk failure would have killed the array.”

The four techniques that produce the biggest delta

These are the four I landed on after a weekend of reading forum posts, mailing list archives, and the mdadm source. They are not the only knobs you can turn, but they are the ones that produce the largest change for the smallest risk.

1. Leave bitmaps on

Bitmaps let mdadm track which stripes (the chunks of data that mdadm writes across the array, typically 512K or 1MB each) are in sync, so a resync only rebuilds the parts of the array that actually changed. With bitmaps on, a typical rebuild after a clean shutdown or a known-good disk swap can be orders of magnitude faster than a full resync. There is essentially no reason to turn bitmaps off on a modern system.

Verify the bitmap is present and active with:

mdadm --detail /dev/md0 | grep -i bitmap

If you are assembling the array manually, enable bitmaps explicitly:

mdadm --assemble --bitmap=internal --update=resync /dev/md0

The internal bitmap lives on the array itself, so there is no extra disk to manage. The cost is a small write amplification (every write to the array also touches the bitmap), which on most home workloads is invisible. If you are running a database that does 50,000 writes per second, you would notice it. For a Plex box or a file server, you will not.

2. Raise the resync speed limit

The default speed limit (speed_limit_max) is set to 200,000 KB/sec by default on most distros, which is conservative. You can raise it during a rebuild without affecting normal operation. The kernel will throttle itself if I/O is saturated, so cranking this too high does not actually damage anything, but it does let mdadm push harder when the disks are idle.

To set a faster limit for the current resync only:

echo 1000000 > /proc/sys/dev/raid/speed_limit_max

The unit is kilobytes per second. A value of 1,000,000 (1 GB/sec) is a reasonable upper bound for SSDs and a fast SATA SSD array. For spinning rust, somewhere between 200,000 and 500,000 is the realistic ceiling. Set it too high and the kernel will simply throttle itself down to the disk’s actual throughput, so the only cost of a too-high number is that you set yourself up for disappointment.

To make the change survive a reboot, add it to /etc/sysctl.conf:

dev.raid.speed_limit_max = 500000
dev.raid.speed_limit_min = 50000

The speed_limit_min is the floor that mdadm will not drop below, even when the system is busy. Default is 1,000 KB/sec, which is glacial. Setting it to 50,000 means the resync always makes meaningful progress.

3. Read-ahead and stripe cache tweaks

The block layer (the part of the Linux kernel that mediates between software and the storage devices) has a read-ahead setting for each array, which controls how much data the kernel pulls into memory ahead of the current read position. Higher read-ahead means the resync process can stream through the array faster on spinning disks, because it is not waiting for each individual block to come back from the drive before requesting the next one.

Check the current value with:

blockdev --getra /dev/md0

For a typical home server, 4096 or 8192 sectors is a sensible value. The default is usually 256, which is too small for sequential workloads like a resync. You can change it with:

blockdev --setra 8192 /dev/md0

This is a runtime change and does not survive a reboot. To make it persistent, add a udev rule (a small file in /etc/udev/rules.d/ that the kernel applies when it sees a new storage device):

# /etc/udev/rules.d/99-md-read-ahead.rules
ACTION=="add", ENV{ID_FS_TYPE}=="linux_raid*", RUN+="/sbin/blockdev --setra 8192 %k"

The udev rule fires every time the array comes online, so the setting is correct after every boot, every disk replacement, and every mdadm --assemble.

4. Stop services that fight for the disks during the rebuild

This is the unsexy one and the one nobody mentions. If your Plex server is transcoding a 4K stream while your array is rebuilding, the resync throughput will collapse, because every disk seek for the transcoding job is a seek the resync has to wait for. The fix is not to disable Plex. The fix is to schedule the resync for a time when the disks are idle.

You can pause and resume a resync with:

echo idle > /proc/sys/dev/raid/speed_limit_min

Setting speed_limit_min to idle tells mdadm to only resync when the disk would otherwise be idle. This is the gentlest of the four techniques, and it is the right move when you have a server that needs to keep serving during a rebuild.

For servers where downtime is acceptable, the brute-force version works too: stop the services, run the resync with the aggressive speed limits from technique 2, and start the services back up. The rebuild finishes much faster because there is no competing I/O. The trade-off is that whatever depends on the array is unavailable during the rebuild.

Before any of these tweaks, run through this short checklist to make sure you are not about to make things worse:

• Verify the array is clean. cat /proc/mdstat should show [UU] for a mirror (both disks up) or [UUUU] for a RAID 6, not [_U] or [U_U]. If a disk is marked failed, fix that first.
• Check the SMART data on every disk. A resync on a disk that is already failing can push it over the edge. smartctl -a /dev/sdX and look at Reallocated_Sector_Ct and Current_Pending_Sector. Anything non-zero is a yellow flag.
• Confirm you have backups. A resync is the most I/O-intensive operation your array will see in a normal month. A second disk failure during a resync is a real risk. Backups should be on a separate device, not on the same array.
• Note the current rebuild time estimate. cat /proc/mdstat shows ETA (estimated time remaining) for any active resync. Write it down. If the new techniques do not produce a faster ETA within 10 minutes, something is wrong.

What I would tell past me

If I could send a message back to the version of me that started the 14-hour rebuild last weekend, I would say three things.

• Bitmaps are not optional. If your array does not have an internal bitmap, add one before you do anything else. A clean shutdown plus a missing bitmap is the difference between a 3-hour rebuild and a 14-hour rebuild.
• speed_limit_min matters more than speed_limit_max. The default minimum is 1,000 KB/sec, which means the resync crawls when the system is busy. Raise the minimum to 50,000 or 100,000, and the rebuild keeps making meaningful progress during normal operation.
• Pause, do not cancel. If you need to use the array urgently, set speed_limit_min to idle. The resync stops fighting for the disks, your services work normally, and the rebuild resumes the moment you are done.

Trade-offs

Pushing resync speed is not free. The four techniques above buy you a faster rebuild at a real cost. Higher speed_limit_max means a louder array on spinning disks, because the heads are seeking constantly. Bitmaps add a small write penalty on every write to the array. Aggressive read-ahead pulls more data into the page cache (the kernel’s in-memory buffer for disk reads), which can starve other workloads for RAM. And pausing instead of letting the resync run during peak hours extends the total time the array is in degraded mode.

In our case, the 3-hour rebuild on a quiet Saturday morning was the right trade. The same tweaks during a weekday, with Plex transcoding for two kids and a backup job running, would have produced visible stuttering on the streams. Your math will be different if your workload is heavier than mine, or if your disks are slower than mine. A pair of 7200 RPM SATA drives has a different ceiling than four NVMe SSDs in a RAID 10 (a striped mirror: data is mirrored across pairs, then those pairs are striped for speed).

The migration cost for the four tweaks is about 30 minutes for someone comfortable with mdadm, including reading the man page. The biggest gotcha is the read-ahead setting not surviving a reboot without the udev rule, which is the kind of thing you only notice the next time you rebuild.

If you have SSDs and a non-critical workload, this is a clear win: push every knob to the max and let the array rebuild in minutes, not hours. If you are running a heavy database on spinning rust during business hours, the right move is the opposite: lower speed_limit_max, set speed_limit_min to idle, and let the rebuild take 24 hours instead of fighting for the disks.