I have a confession. The first time I ran a word-frequency pipeline against a man page, I shipped numbers that were off by a factor of three. The output looked plausible. The top word was “the” at 267, which is exactly what you would expect. The problem was the second row, which reported 7,702 empty strings as the most frequent “word” in the file. I only caught it because I checked the tail of the output instead of trusting the head. That experience is the entire reason I treat copy-pasted shell one-liners as guilty until proven innocent, and it is the entire reason this article exists.
Most of the word-counting pipelines floating around the web are 20 years old. They split on a single space, try to clean up afterwards, and quietly produce wrong numbers on modern UTF-8 (Unicode encoding that uses 1-4 bytes per character) text. The list of tools involved has not changed: wc (word count), tr (translate characters), sort, uniq (collapse duplicates), fold, grep (pattern search), and awk (pattern scanning and processing). What has changed is the input. UTF-8 is the default locale (language/region encoding setting) on every current Linux distribution. The locale is rarely C. Sloppy pipelines that worked on a 2014 ASCII man page will hand you nonsense today, and the nonsense will look close enough to right that nobody will notice.
Ubuntu 26.04 adds another twist. The default userland (the set of utilities available from the shell) for wc, sort, uniq, fold, tr, and head is now rust-coreutils, a Rust reimplementation of the GNU core utilities, shipped at version 0.8.0 in the Ubuntu release. GNU coreutils is still installed as a compatibility fallback. The two implementations are not identical, and a few of the pipelines below can return slightly different counts depending on which one is providing the command. grep and awk are not part of this transition, so they are safe building blocks when character handling matters.
The pipelines that actually work
Before you trust a character count, find out which implementation of wc your shell is using. The simplest check is to ask the binary directly:
$ wc --version
wc (GNU coreutils) 9.7
If you see “rust-coreutils” instead, the patches below still work, but a handful of the outputs (specifically, the comparison between -c and -m on multi-byte text) may differ at the byte level. The shape of the results holds. The exact numbers belong to whichever box produced them.
For the examples here I used the man page for man itself, which is conveniently installed everywhere and contains ordinary English prose plus command names plus formatting noise. If man reports a missing page on Ubuntu or Debian, install it with sudo apt install man-db manpages. On RHEL-family minimal images, /etc/dnf/dnf.conf often sets tsflags=nodocs, which discards documentation at install time. Comment that line out and reinstall the package before the man pages will actually land on disk.
The clean version of the ten most frequent words, in a single pipeline, looks like this:
$ grep -oE '[[:alpha:]]+' man.txt | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -rn | head -n 10
Reading it stage by stage: grep with -o (only match) prints each run of alphabetic characters on its own line, throwing away punctuation and whitespace; tr lowercases everything so “Man” and “man” count as the same word; sort groups identical lines together, which is what uniq requires; uniq -c collapses each run and prefixes a count; the final sort -rn ranks numerically in reverse order; head takes the top ten. Each of these tools is older than most of the engineers who use them, which is part of why they still work.
The classic buggy version is one character away from correct: it splits on a single space with tr ' ' '\012', then tries to strip punctuation and blank lines afterwards. The problem is that manual pages are formatted with runs of spaces, so splitting on one space turns each run into multiple empty lines. A trailing grep -v '[^a-z]' is supposed to filter them out, but an empty line contains no character that is not a lowercase letter, so it passes straight through. The fix is to extract words first and never split on a single space in the first place. That is what grep -oE '[[:alpha:]]+' does, and it is why the new pipeline is also faster: it does not generate the millions of empty tokens the old one has to throw away.
A single-pass awk version that builds the frequency table in memory is shorter to type and slightly faster on small files:
$ awk '{ for (i = 1; i <= NF; i++) { w = tolower($i); gsub(/[^a-z]/, "", w); if (w != "") freq[w]++ } } END { for (w in freq) printf "%7d %s\n", freq[w], w }' man.txt | sort -rn | head
awk’s NF holds the number of fields on the current line, so the for loop visits every whitespace-separated field. tolower converts each one, gsub strips non-letters, the empty check skips junk, and freq[w]++ increments the in-memory count. The sort at the end is still required for ranking, so this approach does not eliminate sorting altogether; it just moves counting into awk’s associative array (an in-memory hash table) and avoids the separate sort+uniq stage. Counts can differ slightly from the grep version because awk splits on whitespace first, so “man(1)” becomes “man” and “read/write” becomes “readwrite”. Pick whichever definition of “word” matches what you are actually measuring.
Where the rest of the magic lives
fold is the underrated tool in this set. fold -w1 breaks input into one-character lines, which makes it a convenient way to inspect characters one at a time or feed them into another pipeline. For column-aligned display, tr can swap characters: tr ' ' '_' replaces spaces with underscores, tr -d '\r' strips carriage returns that show up when you process Windows text files. For finding lines you care about, grep with -c counts matches instead of printing them, and -l lists filenames instead. For more structured work, awk’s associative arrays handle the bookkeeping for you without a separate counting stage.
The four tools that come up in ninety percent of useful text pipelines, with the one flag I would not skip on each:
- grep -o. Extract tokens instead of full lines, so punctuation does not pollute your counts.
- tr ‘[:upper:]’ ‘[:lower:]’. Locale-aware case folding; the bracket classes handle UTF-8 correctly when the locale is set.
- sort -rn. Numeric reverse sort for ranking; without
-nyou get alphabetical sort of the counts, which is wrong. - uniq -c. Counts adjacent duplicates after sort; useless without a sort in front of it.
If you only remember three flags from this whole article, make them these. -o on grep to extract tokens instead of lines, -m on wc to count characters in your current locale (and explicitly set LC_ALL=C.UTF-8 if you care which locale you get), and -w on grep -v to do whole-word exclusion so your stop-word list does not silently eat substrings out of real words. Those three flags cover about ninety percent of the mistakes I have seen in pipelines people ship to production.
Trade-offs
Shell pipelines are not the right tool for every job, and three things bit me badly enough to remember them.
Locale-dependent output is the first. The same pipeline can return different character counts depending on the active locale, and the difference is invisible unless you look for it. Set LC_ALL=C.UTF-8 (or whatever UTF-8 locale your distribution ships) explicitly at the top of any pipeline where character-vs-byte matters. Do not assume the shell you inherited has the locale you want.
Performance on large files is the second. The grep-tr-sort-uniq-sor pipeline is fine on a man page and falls apart on a 10 GB log file because the intermediate sort stages need to hold the working set in memory or spill to disk. For files that big, switch to awk in-memory counting, or reach for a real log aggregator. None of these one-liners are designed for streaming over a 100 GB input.
Platform drift is the third. Ubuntu’s move to rust-coreutils is the visible example, but every distribution ships slightly different defaults for tr, sort, and awk across major versions. A pipeline that works on Ubuntu 22.04 may behave differently on RHEL 9. Test on the actual target system before you trust the numbers, and pin the relevant coreutils package version if the result matters.
If you only run a pipeline once on a small file, none of this matters and the copy-pasted version is fine. If you run a pipeline every week on a file whose output someone reads, the five minutes of explicit locale-setting and version-checking will save you from the kind of “are these numbers right” email that ruins a Tuesday.
Bottom line
The right way to count words on Linux in 2026 is the same as it was in 2006: extract alphabetic tokens with grep, lowercase with tr, count with sort and uniq, rank with sort. The difference is the locale and the userland. Set LC_ALL=C.UTF-8 explicitly, know whether wc is GNU or rust-coreutils on your box, and never split on a single space if you care about the output. That is the whole game.