I Cloned My Repo Three Times Before I Learned This Trick

I have a private monorepo (a single repository holding many related projects that share tooling and dependencies) at work with about 60 GB of generated artifacts checked into git history by a long-departed engineer. The first time I cloned it over our slow office VPN, I sat and waited 47 minutes for the initial fetch. The second time, after I had to recreate my dev environment on a new laptop, I did the same thing and waited 45 minutes. The third time, I sat down and figured out what was actually happening, fixed it, and now the same clone takes about 90 seconds.

The fix is not new. It is not even obscure. It is one of those git features that has been there since 2014 and that almost nobody outside of the kernel and game-dev crowds uses, because the docs assume you already know why you would want it. Once you understand the why, the rest is a single command. I want to write this down because I have had to explain it to four coworkers in the last six months and the explanation is the same each time.

What is actually in a git repo

The thing most people misunderstand about git is that it does not store files. It stores four kinds of objects (a term git uses for its on-disk storage units): blobs (file contents), trees (directory listings), commits (snapshots with metadata), and tags (named pointers to commits). When you clone a repo, git fetches all the objects the remote has, then checks out the tree referenced by the branch you asked for. The objects that make up the tree you actually check out are usually a small fraction of the total objects in the repo.

For most repos, the difference is irrelevant. The objects git fetches but you never look at are a few megabytes, and the network is fast. For some repos, the difference is enormous. The objects git fetches but you never look at are tens of gigabytes, and the network is slow. The first kind of repo is the kind you use at a startup. The second kind is the kind you use at a company with years of accumulated binary blobs.

In my case, the second kind had grown to the point where a fresh clone over a residential VPN took close to an hour. The repo I actually worked in was about 800 MB. The other 59 GB was historical.

What git calls the fix

The feature is called a partial clone (a clone mode where git only fetches the objects you actually read, not every object the remote has). It has been stable in mainline git since 2.17 and works against any remote that speaks the v2 protocol (the modern, more efficient version of how git clients and servers talk to each other). GitHub, GitLab, Bitbucket, and most self-hosted Gitea and Forgejo instances all support it.

The way you turn it on is a single command-line flag at clone time:

git clone --filter=blob:none https://github.com/owner/repo.git

The blob:none filter tells the remote: send me commits and trees, but do not send me blob objects (file contents) until I actually need them. Git will fetch the blob for a file the first time you read it. For most working directories, that means git fetches the blob for whatever file you cat, less, or open in your editor, then caches it locally. The repo you check out is complete. The objects git fetched to your disk are a fraction of what they would be otherwise.

A second filter, --filter=blob:limit=1m, drops blobs larger than 1 MB from the initial fetch entirely. That is the setting I use for the 60 GB monorepo. Almost all of the bulky files in there are generated artifacts that I never read in the source tree. The 800 MB I actually work with fetches in 90 seconds because the rest is excluded.

What I had to relearn

The partial clone has two practical implications that bit me on day one.

First, any operation that walks the history without checking out specific files will be slow the first time. If you run git log --all --stat to see what files changed across the entire history, git has to fetch every blob it needs to display, and you wait. The fix is to scope the command. git log -- path/to/file only fetches the blobs for that file. git log --all --no-renames --diff-filter=A only fetches the blobs for the commits where the file was added, not for every subsequent change.

Second, any operation that touches files outside your normal working set will trigger a fetch. git checkout old-branch to look at a feature branch from 2022 will fetch every blob in that branch. If the branch contains 5 GB of generated artifacts, you wait for 5 GB. The mitigation is git sparse-checkout (a feature that lets you check out only a subset of the working tree, leaving the rest on the remote until you ask for it). Set up a sparse-checkout for the directories you actually work in, and old branches you do not look at often stay cheap.

These two gotchas are not deal-breakers. They are a different way of working with git, and once you internalize them, the partial clone is faster than the full clone in every workflow except “I need every byte of every file in every commit right now,” which is a workflow almost nobody has.

What I would tell past me

Four things, in order of priority:

• Add --filter=blob:none (or --filter=blob:limit=10m for medium repos) to your muscle-memory clone command. Even on a small repo it does no harm. On a large repo it is the difference between a 5-minute wait and a 45-minute wait. The cost is a one-time mental adjustment to remember that some operations are slow until you scope them.
• Set up sparse-checkout for any monorepo where you only work in a subdirectory. The repo I work in has 14 top-level directories. I work in 2. Sparse-checkout means git only materializes the 2 I care about, and the other 12 stay as remote references until I check them out explicitly. Disk usage on my laptop dropped from 4 GB to 400 MB.
• Audit the operations that walk history before you run them on a partial-cloned repo. git log --all --stat and git checkout ancient-branch will trigger bulk fetches the first time. Scoping them to a path or a single ref keeps them cheap. The fetch is invisible, but the wait is not.
• Stop checking generated artifacts into git. I know the engineer who wrote the original monorepo meant well. Binary outputs, build outputs, large data files, anything that can be regenerated belongs in a separate artifact store, not in source control. Git is for source. Artifact stores (Nexus, Artifactory, Git LFS, S3) are for artifacts. The line between them is one of the most expensive lessons in software infrastructure.

Trade-offs

The partial clone is not free. Some git GUI tools do not handle the on-demand fetching well. If you open a GUI that walks the entire repo to build an index, it will be slow until the index is built, and it will request every blob in the process. Sourcetree, GitKraken, and the older versions of the GitHub Desktop app all have varying degrees of partial-clone support. The command line is fine. Fork (the macOS git GUI) is fine. Anything older than about 2020 may struggle.

A second trade-off is that partial clone is more complex to reason about. A full clone is a self-contained unit you can tar up and move to another machine offline. A partial clone has dependencies on the remote that you cannot fully satisfy without network access. If you need an air-gapped copy of a repo for a build environment, partial clone is the wrong tool. Use a full clone, then convert.

A third trade-off is the server side. Some self-hosted git servers, especially older versions of Gitea and some homegrown setups, do not advertise partial clone support in their v2 protocol handshake. When that happens, git falls back to a full clone and prints a warning that is easy to miss. If your --filter=blob:none clone is not faster, the server may not support it. The fix is either to update the server or to switch to a different remote for that repo.

For most individual developers and most small-to-medium repos, partial clone is a clear win. For large monorepos with lots of binary history, it is the difference between a usable dev environment and an unusable one. For air-gapped build systems, it is the wrong tool.

If you have ever waited 30 minutes for a clone over a slow connection, spend the 30 seconds to add --filter=blob:none to your next clone and see how much faster it gets. The feature has been stable for a decade and there is no good reason not to use it.