I run a GeForce RTX 2070 Super on my desk, and it has been quietly handling every local AI workload I have thrown at it for the last six months. The card shipped in 2019, has 8GB of VRAM (the dedicated memory on the GPU that holds the model’s parameters during inference), and would not have been my first choice a year ago for any model heavier than a 7B parameter chat bot. That is no longer true. The shift is not because the GPU got faster. The shift is that a different class of model landed, and those models happen to fit comfortably on the kind of VRAM I already had.
The headline is that I cancelled two paid AI subscriptions in the last quarter, and I have not noticed a quality drop. The catch is that I am not running the same models I was running on those subscriptions. I am running different ones, on a different cadence, and the workflow I have landed on is the point.
What changed in 2026
A few things converged at the same time. The first is the rise of Mixture of Experts (MoE) architectures, which only activate a fraction of the model’s total parameters per token. A 30B parameter MoE model might use 3-4B of those parameters at any given moment, which means the VRAM you need to run it at reasonable speed is closer to the active count than the headline number. That is why an 8GB card that would choke on a dense 13B model can comfortably host a 30B MoE.
Quantisation is the second change, the process of reducing the precision of the model’s weights from 16-bit floats to 4-bit or 8-bit integers. That roughly halves the memory footprint per parameter for each step down the precision ladder. A 4-bit quantised 8B model fits in under 6GB of VRAM, and the quality loss compared to the full-precision version is small for most chat and summarisation workloads.
Tooling is the third change. llama.cpp, Ollama, LM Studio, and a dozen smaller projects now handle model download, quantisation, and serving with a single command. The plumbing that used to require a CUDA-aware Python environment and a custom inference server is now a one-line install.
All three together is what made my 2070 Super useful again. Any one of these would have helped. All three at once is the inflection.
The setup that actually works
The hardware is a desktop with a Ryzen 7 5800X, 32GB of system RAM, and the aforementioned RTX 2070 Super. The OS is Ubuntu 24.04 LTS. The inference stack is Ollama running as a system service, with Open WebUI as the front end on a different machine on the same LAN.
The model lineup that I have settled on is small and specific:
- Qwen 2.5 14B (Q4_K_M quantisation). General chat, summarisation, draft writing. This is the workhorse.
- Mistral Small 22B (Q4_K_M). Heavier reasoning tasks, code review, longer context. Slower but more accurate.
- Llama 3.1 8B (Q8_0). Quick classification and routing tasks where latency matters more than depth.
- Nomic Embed Text v1.5. Embeddings for a personal semantic search index.
I keep all of them on a 1TB NVMe drive. Total disk usage is about 65GB, and I have not run out yet.
The two paid subscriptions I cancelled were ChatGPT Plus and Claude Pro. The work I was using them for, which is mostly writing help, code review, and summarisation of long documents, is now split between Qwen 2.5 14B and Mistral Small 22B. I have not gone back.
What I actually use it for
About 80 percent of my local AI usage falls into four buckets.
- Drafting. I write the first 200 words of an article, then ask Qwen to extend or compress. The output is rarely usable as-is, but it gives me a second opinion in 20 seconds without leaving the network.
- Code review. Mistral Small 22B with a 32K context window is good enough to catch the dumb mistakes in a 200-line pull request. It is not as good as Claude Sonnet on a hard refactor, but for the routine work it is more than adequate.
- Summarisation of long documents. I drop PDFs and Markdown files into Open WebUI and ask for a 200-word summary. For meeting notes, vendor contracts, and 60-page technical whitepapers, this saves me an hour a week.
- Embedding search. I run a private semantic search over my own notes. Nomic Embed gives me good enough recall, and the index never leaves my network.
The other 20 percent is experimentation. I try new models when they drop on Hugging Face, I run benchmarks against my old ChatGPT exports to see if quality has regressed, and I occasionally try weird combinations (a small chat model chained to a small code model, for example). The experimentation is what keeps the setup from going stale.
Trade-offs
Local AI is not free in time. The first cost is the model selection churn. New MoE checkpoints drop every few weeks, and the quality differences between them are real but not obvious without testing. I spend maybe two hours a month running the same 30 prompts against a new model and the old one, and updating my lineup based on the results. A paid subscription is zero time on this axis.
The second cost is context length. Mistral Small 22B at Q4_K_M handles 32K context on my hardware at about 8 tokens per second. That is fast enough for reading but slow enough that a 100K context document takes real wall-clock time. Cloud models are faster on the same task, and frontier models are faster still.
The third cost is the occasional hard problem. There are prompts where a local 22B model loops, hallucinates, or simply gives up. On a paid subscription I would have escalated to a bigger model. Locally, the only escalation is to wait for the next model release. For most of what I do, that is fine. For the work where it is not, I have not found a clean workaround yet.
In our case, the time I save on the routine work outweighs the time I lose to model churn and the hard problems. Your math will be different if your workload is mostly the hard problems, or if you are not comfortable rebuilding a quantised model from a Hugging Face checkpoint when the upstream repo changes its name.
Bottom line
If you have an NVIDIA card from the Turing generation or newer with at least 8GB of VRAM, you have more local AI capability than the public discourse suggests. The Turing line launched in 2018, so a 7-year-old card is exactly the kind of hardware that was written off for AI two years ago. The combination of MoE architectures, 4-bit quantisation, and mature inference tools has closed the gap, and the gap is now narrower than the marketing claims from cloud vendors would have you believe.
Bottom line: if your work is mostly writing, code review, and summarisation, a $0 local stack on hardware you already own will handle 80 percent of it. The last 20 percent is where the cloud still earns its price tag, and pretending otherwise would be dishonest. For most of what I do, the local setup is the right call.