>
Uncategorized

Sending a 10 GB file without renting someone else’s cloud

I have been the person on both ends of the 10 GB file problem more times than I want to admit. The last time was a video project, hand-off to a collaborator, no way to use the same Dropbox account. We tried WeTransfer. We tried Google Drive. We tried a USB drive in the mail, which arrived corrupted. The fix was a tool called Wormhole, by the team that makes Magic Wormhole, and I have been quietly recommending it to anyone who asks ever since. It sends files directly between two computers, encrypted end to end, with no server in the middle storing the bytes. The 10 GB limit is not a marketing claim. It is a real limit, and the protocol is designed to handle it on consumer hardware.

If you have ever sworn at a file-transfer service for being slow, expensive, or suddenly requiring a paid tier, this article is for you.

What Magic Wormhole actually is

Magic Wormhole is a small protocol and a small command-line tool. The protocol’s job is to negotiate a connection between two computers, exchange a one-time code, and ferry bytes from one to the other. The tool is a Python package called magic-wormhole that ships a command called wormhole. You run wormhole send filename.zip on one machine, it prints a short code, you run wormhole receive on the other and type the code, and the file transfers. The relay server only sees encrypted blobs. It cannot read the file, and it does not store a copy.

A small public server run by the same team operates the relay. It is the only piece of infrastructure in the path, and its only job is to ferry the encrypted blobs between the two endpoints. Once the endpoints have negotiated a session, the actual data transfer can go directly between them, with the relay out of the loop. Self-hosters run their own copy of this relay because the code is open source and the server is small enough to run on a Raspberry Pi.

Designed around the assumption that the two endpoints are on different networks, the protocol runs behind different NATs (Network Address Translators, the layer of your home router that gives your laptop a private IP and shares one public IP across your whole household), and that neither side has a public IP address. The relay handles the rendezvous (the part where two computers that do not know each other find a way to talk) and the NAT traversal (the part where two computers behind home routers find a way to talk directly to each other), and the data goes directly between the endpoints once the connection is established. This is the same trick WebRTC uses for video calls, and it is the reason Magic Wormhole does not need a server with a 10 GB hard drive.

Real and intentional, the 10 GB limit reflects the protocol design. The transit relay buffers a small amount of data, and the buffering strategy is optimized for small to medium files. For a 10 GB file, the tool splits the file into pieces and transfers them in order. The pieces are small enough that the relay can hold them in memory, and the protocol uses forward error correction so a dropped packet does not require a full retransmit. The end result is that a 10 GB file transfers at roughly the speed of the slower of the two internet connections, with no real overhead.

How I used it on the video project

Setup for the video handoff was a 4K master file, about 8.7 GB, with the collaborator in a different city. I started the send side on my laptop with wormhole send master.mov. The tool printed a code, something like 7-crossover-clockwork. I sent the code over Signal. The collaborator ran wormhole receive on their desktop, typed the code, and the transfer started.

Twenty-two minutes is how long the transfer took over their home connection, which is roughly what I would expect for an 8.7 GB file over a 70 Mbps uplink. My end was on a 200 Mbps connection, so my side was waiting most of the time. Progress shows on both ends, which is helpful for a transfer that long. The collaborator’s machine verified the file hash automatically on completion, and we were done. No account creation, no subscription, no upload to a cloud service that might be down or might decide to throttle me next month.

A year earlier, the same project took three days because WeTransfer was being slow, then Google Drive was being slow, then the USB drive failed. Twenty-two minutes were the result of a tool that is doing less, not more. The 8.7 GB went directly from my laptop to the collaborator’s desktop, with the relay only handling the connection setup and a small amount of buffer.

What about the file size limit

Designed for files up to about 10 GB, the Wormhole protocol will warn you if you go over. The reason is the transit relay’s buffer and the way the protocol handles flow control (the mechanism that prevents a fast sender from overwhelming a slow receiver). For larger files, the recommendation is to split the file with split or zip first, send the parts, and reassemble on the other end. The tool supports --transit-helper for this, but the manual split is fine for the one-off case.

I have not personally tested the 10 GB limit. My largest real transfer was the 8.7 GB video file, and that worked. The protocol’s design suggests 10 GB is a soft cap, and the tool will sometimes push past it with a warning. For files larger than 10 GB, the right tool is rsync over a Tailscale connection (a virtual private network that gives your machines stable IPs and lets them talk to each other as if they were on the same LAN) or a direct scp (secure copy) over SSH. Magic Wormhole is for the awkward middle case, where the two endpoints are on different networks, neither has a public IP, and the file is too big for email but too small for a real file sync.

Resume is another thing the tool does not do well. If the transfer fails partway, you have to start over. For a 10 GB file on a flaky connection, this is a real cost. The protocol has a resume mechanism in the design, but the tool does not implement it yet. For transfers that might take an hour, the workaround is to use a file sync service that supports resume, or to split the file into smaller parts and send them individually.

What I would tell past me

If I could send a message back to the version of me that spent three days trying to send a video file in 2024, here is what I would say:

  • Install wormhole on both machines before you need it. The first time you need to send a big file, you do not want to be installing a Python package and a relay client. Install it on every machine you use, and the next time the need comes up, the tool is already there.
  • Use Signal or iMessage for the code, not email. The code is a one-time secret. Email is a one-time secret in a place that is going to sit in someone’s inbox for ten years. Signal is the right tool for a one-time secret.
  • Do not start the send before the receive is ready. The protocol is symmetric, and the code is single-use. If you start the send and the receive side is not ready, the code is wasted and you have to start over.
  • Pick a relay before you need one. The default relay is reliable, but for the paranoid case, having your own relay already configured saves the day the public one is down. A 30-line config file on a small VPS is enough.

Trade-offs

Magic Wormhole is not a replacement for cloud storage. It is a tool for sending a file from point A to point B, and the file is not stored anywhere after the transfer completes. If you need a backup, a sync layer, or a way to share a file with someone who is not at their computer right now, this is the wrong tool. Use a cloud service for that.

The relay is the other trade-off. The default relay is run by the same team that makes the tool, and it is free. The team has been good about keeping it up, but the relay is a single point of failure for the connection setup. If the relay is down, the tool does not work. The fix is to run your own relay, which is a 5-minute job on any small VPS, and then the tool is dependent on your own infrastructure. For most users, the default relay is fine. For paranoid users, the self-hosted relay is a 30-line config file.

Time is the first real cost. The protocol is fast, but the time is still bounded by the slower of the two internet connections. For a 10 GB file over a 50 Mbps uplink, that is 27 minutes. The user has to be present at both ends for the transfer to complete. Money is the second cost: the tool is free and the relay is free, but the electricity and the bandwidth have a real cost. For a 10 GB transfer on a metered connection, that might be a few dollars. Features are the third cost. The tool does not do resume, does not do multi-file sync, and does not do cloud backup. It does one thing, and it does it well. Upside is real too. The protocol is end-to-end encrypted, the relay cannot read the file, the file is not stored anywhere after the transfer, and the tool is open source and auditable. For a 10 GB file with sensitive content, that is the right set of properties.

If you send large files between computers on different networks, and you want the transfer to be encrypted and not stored on a third-party server, Magic Wormhole is a clear win. If you need a backup or a sync layer, this is not the tool, and you should look at Syncthing or a real cloud service. If you are paranoid about the relay, run your own. The setup is a 30-line config file and a small VPS.

Bottom line

Magic Wormhole is the kind of tool that I wish I had known about five years ago. It is small, it is fast, it is encrypted, it is open source, and it solves a real problem that the cloud services are not great at solving. The 10 GB file size limit is a feature, not a bug, and the design reflects that. Install it on every machine you use, and the next time the need comes up, the tool is already there.

Leave a comment