Hunting duplicate files from inside Ranger’s filter stack

Ranger (a terminal file manager for Linux) has a feature most people who use it have never touched: the filter stack. It is one of those parts of the program that sits quietly in the background until the day you have a folder of three hundred files and you want to know which ones are duplicates. Then it becomes the only thing that matters.

The man page describes the filter stack in about four lines and shows one example that almost works. The example is enough to see what the feature is for and not enough to actually use it. This is the article I wish had existed when I was learning it.

What the filter stack is

Ranger keeps a stack of filters between you and the files. Each filter trims the visible list. With no filters active, you see everything in the current directory. With one filter, you see only what passes it. With several, you see only what passes all of them, joined by the connectives you choose.

Three kinds of filter do the actual work:

• Type filters. These match on what a thing is. type f matches regular files. type d matches directories. type l matches symbolic links.
• Name filters. These match on the basename. name *.log matches anything ending in .log. Case sensitive by default.
• Attribute filters. This is where the interesting one lives. duplicate matches files whose content hashes are not unique within the current view.

That last one is the duplicate filter. It works by hashing every visible file’s contents and keeping only the ones whose hash appears more than once in the set. Files whose hash appears once get hidden. The hash is the SHA-1 of the file contents, which is fast enough for a few thousand files and not so fast that you want to run it on a million.

What “duplicate” actually means

The word is misleading. The duplicate filter does not match files that are duplicates of files outside the current directory. It matches files that have duplicate content among the files currently visible.

Practical upshot: if you open Ranger in a folder with one hundred text files and ten of them are identical copies of each other, the duplicate filter shows you those ten and hides the other ninety. If the same identical file exists somewhere else on the disk but not in this folder, the filter does not care. It is a local operation against the current view.

Practical upshot number two: the filter needs every file in the current view to be hashed. On a folder of large video files this is slow the first time and fast on subsequent runs because Ranger caches the hash by inode and mtime. On a folder of small text files it is fast every time.

How to use it

The example in the man page is almost right and has one typo. The intended sequence is:

:filter_stack add type f
:filter_stack add duplicate
:filter_stack add and

Three lines. First, restrict the view to regular files. Second, add the duplicate filter. Third, combine them with and so only files matching both conditions remain visible.

Run those three commands and your folder of one hundred files suddenly shows you the ten that share content. Press dd on the ones you do not want. Or mark them with (space), then bulk-delete with :delete.

There are shortcuts. The man page mentions mapped keys for the same operations:

• .f adds type f to the filter stack.
• .d adds type d.
• .l adds type l.
• .n adds name.
• . " adds duplicate.
• . & adds and.
• . | adds or.

So the same sequence using key shortcuts is .f . " . &. Five keystrokes, no colon commands. That is the version I actually use.

What trips people up

Three things trip people on the first attempt. All three are in the man page if you read it carefully, but the example glosses over them.

Strings must be encoded before hashing. If you type :filter_stack add duplicate and get an error that mentions string encoding, your version of Ranger is too old. The duplicate filter landed in 1.9.3 and the string-encoding fix landed in 2.0. Anything older than that prints an error instead of working. Upgrade, or build from source if your distro is stuck.

The filter is local to the directory you opened it in. If you navigate into a subdirectory, the filter stays active but it only sees the new directory’s files. That is the design. If you want to scan duplicates across the whole tree, open Ranger at the parent and use S (capital) to enter a flat-directory mode where every file in the subtree is listed in one view. The duplicate filter then operates on the flat list.

The filter only sees regular files by default. If you have symbolic links pointing at duplicate content, the link itself is not flagged as a duplicate unless you also add type l to the stack. The combined stack :filter_stack add type f + :filter_stack add type l + :filter_stack add duplicate + :filter_stack add or shows duplicates among either regular files or links. That is the variant from the man page example and it is what most people actually want.

When the filter stack is the wrong tool

There is a different tool for a different problem. If your goal is to find every duplicate file on a disk regardless of directory, you want a content-addressed finder like fdupes, rdfind, or jdupes. Those tools walk the entire filesystem, hash every file, and report collisions. They take minutes on a large tree. The filter stack in Ranger takes seconds on a single directory and is interactive.

There is also a different tool if your goal is to deduplicate by name. The filter stack does not care about names. Two files called report.txt with different contents are not duplicates under this filter. For name-based deduplication, find with a quick basename sort is the move.

What I would tell past me

If I could send a message back to the version of me that spent twenty minutes guessing at the filter syntax, I would say five things:

• The duplicate filter is duplicate, not dup or hash or sha. Just duplicate.
• You need type f first. Without it, you get errors or you get matches against directories, which is not what you want.
• The keystroke is . ". That is period, space, double-quote. Three keys.
• The filter is per-directory. Navigate away and it still applies, but only against what you can see.
• For whole-disk scans, use fdupes -r. The filter stack is the wrong tool for that and the man page does not say so.

Trade-offs

Interactivity is both the strength and the limitation. You can see the duplicates and decide in the moment which to keep. A folder of fifty thousand files will take long enough to hash that you will close Ranger and reach for fdupes before it finishes.

Hash speed comes from SHA-1, which is fast. SHA-1 is also collision-vulnerable in theory, though producing a deliberate collision against a file you actually care about is still well beyond the threat model of “I have three copies of the same notes.md.” For most personal use, SHA-1 collisions are not the worry. For forensic-grade deduplication on untrusted input, use a tool that hashes with BLAKE2b or SHA-256 instead.

Symbolic links need explicit handling. If your home directory has a dotfiles symlink farm pointing at the same files you can see directly, the filter sees them as separate files unless you add type l to the stack. That is fine, but it means a df -h showing “lots of duplicates” can sometimes be the same file counted twice through different paths.

Cache invalidation follows inode and mtime. Rename a file, edit it, save it back, and the cache is invalidated. You rehash. That is the design and it is the sensible one.

Bottom line

If you have a folder where you suspect duplicate files and you want to clean it up by hand, Ranger’s filter stack is the fastest interactive way to do it. Three commands, five keystrokes if you use the shortcuts, and you have a directory showing only the duplicates. From there it is bulk mark, bulk delete, done.

If your problem is bigger than a single folder, or if you want a non-interactive scan, do not use the filter stack. Use fdupes -r or rdfind and pipe the output into a deletion script. The filter stack is a precision tool, not a sweep tool.