Inference is where most agent projects hit a wall, and the wall is not a model problem. It is a serving problem. The freeCodeCamp walkthrough by Darsh Shah on scaling LLM (large language model) inference for AI agents using vLLM is the cleanest practical explanation I have read this month, and it lines up with what I have seen when moving a small agent off the single-user prototype path. The piece is technically careful, which is what you want when the question is whether to adopt a runtime you have not touched before.
vLLM (an open-source inference server, originally from UC Berkeley) is not the only answer for serving agents at concurrency, but it is the one that shows up most in production writeups for a reason. It exposes an OpenAI-compatible API (a programming interface shaped like OpenAI’s HTTP endpoints, so existing OpenAI client libraries work against it with only a base URL change), manages batching and KV-cache (key-value cache, the per-token attention memory the model keeps while generating) memory under the hood, and on Apple Silicon it now has a community-maintained MLX (Apple’s machine-learning framework) backend called vLLM-Metal that lets you run the same workflow on a Mac. I worked through the install and the OpenAI-compatible agent connection on a MacBook Pro with 32 GB of RAM. The whole loop, from server start to first successful agent reply, took about twenty minutes including a wrong-venv mistake on my end.
The reason an agent workload is different
A single agent talking to a single model is the easy case. One request in, one prompt tokenized, one response streamed back. The model weights fit in memory, the response is short, and the bottleneck is token-by-token generation, which is sequential by definition. You can run that on almost anything, and most early agent tutorials do.
A real agent workload is not one request. It is a tree of requests. A single user prompt may trigger a planning call, a tool-selection call, a tool-execution call, an interpretation call, and a final synthesis call. The freeCodeCamp piece cites “10 to 30 separate LLM calls” per user interaction as a representative range, and that number matches what I have seen on production agent traces. Multiply by the number of users and you stop thinking about single requests. You start thinking about scheduling.
The other thing that changes is request shape. A planning call is short. A tool-result interpretation call may carry thousands of tokens of retrieved context. A final response is medium-length but needs to be coherent with everything that came before. Requests arrive at different times, have different input lengths, and finish at different times. The serving layer has to keep all of that efficient at the same time, which is the problem vLLM solves.
What vLLM actually does differently
Two design choices are doing the work, and both of them show up in the source. The first is continuous batching. A traditional serving loop fills a batch of requests, runs them until they all finish, and only then starts the next batch. With continuous batching, when one request finishes, the server slots a new one into the next execution step without waiting for the rest of the batch. This alone changes throughput significantly under uneven workloads.
The second is PagedAttention (vLLM’s block-based memory management for the KV cache). Without PagedAttention, the KV cache for each request needs a single contiguous block of memory, sized to the longest expected output. When requests have different lengths, you waste memory and end up throttling concurrency to avoid running out. PagedAttention splits the KV cache into fixed-size blocks (the same idea as virtual memory pages) and allocates them on demand. Memory fragmentation drops, freed blocks are reusable, and the server can hold more concurrent requests in the same GPU memory budget.
Prefix caching (a feature that lets the server reuse the KV cache for the shared leading portion of prompts, so identical system prompts and tool definitions do not have to be re-tokenized across requests) is the third feature the piece highlights. For an agent that uses the same system prompt and the same tool definitions across every request, prefix caching means the prefill (initial prompt-processing) phase only runs once per session. That is the most concrete win in agent workloads, where the prompt leading line is often hundreds of tokens of tool descriptions.
The OpenAI-compatible API is the practical reason the rest of the agent toolchain works with vLLM out of the box. Existing frameworks, including most agent libraries, have OpenAI clients. You point the client at http://localhost:8000/v1, pass any string as the API key, and the rest of the agent code does not change. The piece walks through the exact Python client setup. I tested it against a tiny agent I had been running against a hosted API. The only diff was the base_url line and the api_key="NA" placeholder. Everything downstream worked.
Running it on Apple Silicon
Standard vLLM installs target Linux with NVIDIA GPUs. The piece uses vLLM-Metal, which is a community-maintained variant that uses Apple’s Metal graphics framework and MLX underneath. The install is a one-line shell script (curl -fsSL ... | bash), the venv (Python virtual environment) is created in ~/.venv-vllm-metal, and the server starts with a command like vllm serve mlx-community/Qwen2.5-0.5B-Instruct-4bit --host 127.0.0.1 --port 8000.
The Qwen 2.5 0.5B model is small enough that the demo runs on a 32 GB Mac without an external GPU. The piece is honest that production workloads need real GPU memory and CUDA (NVIDIA’s GPU compute platform) for full throughput, but the small-model local loop is genuinely useful for development. I could iterate on prompt changes without burning hosted-API credits and without waiting for cold starts.
The vllm serve output tells you when the server is ready: “Application startup complete” followed by the HTTP endpoint. From there, curl http://localhost:8000/v1/models returns the model’s metadata, including the max_model_len of 32768 tokens. That is the same shape as the OpenAI /v1/models endpoint, which is exactly why the OpenAI Python client works against it without changes.
FreeCodeCamp’s walkthrough ends with a small agent that calls client.chat.completions.create(...) against the local vLLM server, asks a simple question, and prints the response. The piece is straightforward enough that I could copy the code into a fresh file, replace the model name with one I had on disk, and have an end-to-end agent in under five minutes. That is the bar. The value of a tutorial is whether you can copy it and have it run; this one clears that bar.
When vLLM is the right choice
The piece is honest about the threshold: vLLM pays for itself when you move past single-user prototype and start handling concurrent, uneven workloads. For a script that makes one API call per minute, you do not need any of this. For an agent that fans out into multiple model calls per user request, with multiple users hitting it at once, vLLM is the kind of change that moves you from “the agent is slow at peak times” to “the agent keeps up.”
The other threshold is platform. vLLM is a real serving system. It needs memory, it needs an accelerator (GPU, Metal, or in some cases a fast CPU with the right backend), and it needs to be running while the agent is running. That is a different operational shape than a hosted API call, which is one HTTP request and no state. If your agent is a serverless function or a one-shot CLI tool, hosted APIs are still simpler. If your agent is a long-running service that handles sustained traffic, the operational cost of running vLLM is the right trade.
On what is not in scope, the piece is honest. The walkthrough does not cover multi-GPU scaling, distributed inference, or fine-tuning (continuing the training of an existing model on your own data). Those are real production concerns and they each deserve their own writeup. For the “I have a small agent and I want to know whether I should run the model myself” decision, the piece is the right starting point.
Trade-offs
Running vLLM locally is not free in operations. The server has to be running while the agent runs, and that means keeping a process alive, watching its memory, restarting it when the model crashes, and watching the logs. Hosted APIs do not have any of that overhead. The trade is cost, latency, and control. A hosted API charges per token and adds a network round trip. vLLM on your own hardware costs electricity and devops attention but skips both.
The model selection is narrower than what a hosted API gives you. vLLM supports a wide and growing list of models, but a hosted API gives you whatever the vendor has shipped that day. If you need the bleeding-edge model on launch day, hosted APIs win. If you need a stable model you can pin and test against for a year, vLLM is the better posture.
Apple Silicon support is a real option, but it is not the same as NVIDIA support. The vLLM-Metal backend is community-maintained, and the model coverage is smaller than the CUDA backend. For local development and small demos, it is excellent. For production workloads with strict latency requirements, NVIDIA on Linux is the path the community has settled on. The piece’s choice to demo on Apple Silicon is the right pedagogical move; the production choice is usually a different one.
What I would tell past me
A few practical lessons from running this end-to-end:
- Start with the smallest model that demonstrates the architecture. The piece uses Qwen 2.5 0.5B. That is small enough to load on a Mac and fast enough to iterate on. A 70B model on the same setup would have spent twenty minutes on model load before showing you anything useful.
- Set up the OpenAI-compatible client abstraction early. Even if you start with a hosted API, write the client code so the
base_urlis a config value, not a hardcoded constant. Switching to vLLM becomes a one-line change rather than a refactor. - Watch the KV cache memory, not the GPU utilization. The piece is right that KV cache is the actual constraint under concurrency. A high GPU utilization with low KV cache usage means you have headroom. A low GPU utilization with high KV cache usage means you are about to throttle.
- Treat the local server as a development tool, not a deployment target. vLLM-Metal is the right answer for “I want to run this on my laptop today.” It is not the right answer for “I want to serve this to ten thousand users tomorrow.” Decide which you need before you commit to the operational shape.
vLLM is one of the few infrastructure pieces that has stayed close to its original design while the surrounding tooling has changed. The continuous-batching, PagedAttention, and prefix-caching choices are still the right choices in 2026, and the OpenAI-compatible API has kept the integration story simple. The freeCodeCamp walkthrough is the kind of practical guide I wish I had two years ago when I first tried to scale an agent past a single user. It is the kind of writeup that should age well.