I have been a heavy Ollama user for about eighteen months. It runs on the Linux laptop, the Mac mini, a small NUC (a tiny fanless barebones PC, basically a board in a box) under my desk, and the home server in a closet. I have Open WebUI in front of it, and I send roughly half my “let me think about this” prompts to a local Qwen model and half to ChatGPT. I tried Jan AI for two weeks to see if I could replace both. I went back to Ollama after fourteen days, and the reason is not the model. The reason is the way the local stack is put together.
Jan AI (jan.ai) is a desktop application that wraps a local model runner with a chat UI, a model library, a Cortex engine for inference, and an OpenAI-compatible HTTP API. It installs with a single AppImage (a self-contained executable you download, mark as executable, and run) on Linux, pulls models from the Hugging Face Hub, supports GGUF (a quantized model format designed to run on consumer hardware), and works on CPU. In short, it is the same product idea as Ollama plus Open WebUI, with the UI and the model runner shipped as a single app. I wanted to like it. I liked parts of it. I stopped using it for the same reason a lot of people stop using local AI tools: the moment I wanted to do something the app did not anticipate, the experience fell apart.
The install story is genuinely better
The first thing Jan AI gets right is the install. I downloaded the AppImage on a fresh Xubuntu 25.04 install, chmod-ed it (set the executable bit via chmod +x), and ran it. The app opened, walked me through a model download for Qwen 2.5 7B Instruct (a 7-billion-parameter language model fine-tuned for following instructions), and I was chatting with a local model in under three minutes. There was no terminal, no ollama pull qwen2.5:7b, no editing of a systemd unit file, no reverse proxy to put Open WebUI in front of the runner. For a user who is not already deep in the Linux local-AI stack, that is a real improvement. The same workflow on Ollama would have meant installing Ollama, installing Open WebUI, configuring the Open WebUI backend to point at Ollama, and opening a browser tab.
The first week of Jan AI was pleasant. The UI is clean, model switching is a dropdown, conversation history is local, and the export-to-Markdown button is one click. I used it for the kind of writing-prompt work I usually send to ChatGPT: summarize this article, draft an email, explain a Go error message. The 7B model handled all of it at about 80% of GPT-4o-mini’s quality, fine for first drafts I am going to rewrite anyway. The local path is offline, free, and private, so 80% quality is a trade I make every day.
Jan’s Cortex engine is a real piece of work. It is the inference backend that Jan AI ships in-process, and it handles CPU offload, GPU detection, and the model loading in a way that “just works” on the hardware I have. I did not have to fight CUDA (NVIDIA’s GPU compute platform) drivers, did not have to pick a quantized build, did not have to think about whether my laptop’s hybrid GPU (a setup with two graphics chips, one integrated for battery and one discrete for performance) was going to route the right way. The first time I ran a model, it used the GPU. The first time I tried a 13B model, it offloaded half to CPU and half to GPU automatically. I have spent hours of my life fighting Ollama on the same hardware, and Cortex just did the right thing.
The first time the experience fell apart
The first thing I tried that Jan AI did not anticipate was loading a model from a custom path. I have a few fine-tunes (custom versions of a base model trained on a specific dataset for a specific purpose) of Mistral 7B that I keep in ~/models/fine-tunes/, and I wanted to load one of them into Jan. The app has a “Model Hub” tab that pulls from Hugging Face, and it has a “Local Model” tab that points at a directory. The Local Model tab expected a directory structure that did not match the way I had organized my files. I moved the files into the structure Jan expected, and the load failed because Jan’s model loader does not support GGUF split files (a single large model split across multiple smaller files to fit download limits, named like model-00001-of-00003.gguf). My fine-tune was a split. Jan’s loader wanted a single file. The app did not give me a useful error. The app did not link to documentation that explained the constraint. The app did not have a “report this as a bug” button in the error dialog. I closed the app, opened a terminal, and started Ollama to see if the same fine-tune would work there.
It worked in Ollama in about ninety seconds. I created a Modelfile (Ollama’s plain-text recipe for packaging a model) that pointed at the split files, I ran ollama create my-finetune -f Modelfile, and I had the model running. The point is not that Ollama is more capable. The point is that Ollama gave me a way to do the thing, and Jan gave me a UI that did not have the affordance for the thing. When you are a power user, the UI is in the way. When you are a beginner, the UI is a gift. Jan AI made a clear choice about which user it is for, and I am not that user.
Four other things I tried in the first two weeks that fell outside what the product expected, in roughly the order I tried them:
- Custom system prompts that exceed the default context. Jan’s system prompt editor accepts a long prompt and silently truncates it. The truncation is not logged. The conversation then produces off-topic answers and the user has no idea why.
- Importing an OpenAI chat history. Jan imports chat history from JSON, but the JSON schema is a Jan-specific schema, not the OpenAI export format. The migration path is “export from OpenAI, write a script to convert to Jan schema, import.” I did not do this.
- Setting up a remote inference endpoint. Jan supports a remote Cortex server, but the documentation for the server is in a separate repo, and the install steps for that repo assume a developer who is comfortable with
pnpmand Node source builds. I did not do this either. - Scheduling a recurring prompt. Jan has no scheduled-prompt or cron-style feature. Ollama does not either, but Ollama exposes a CLI that I can wire into
cron. Jan exposes a desktop app, and a desktop app does not have a cron.
The feature gap that actually mattered
The bigger issue was the API. Jan AI ships an HTTP server that exposes an OpenAI-compatible endpoint, which is the protocol most local AI tools speak. I wired it into my Obsidian vault (a local Markdown knowledge base) using a community plugin, and the plugin started failing after about twenty requests. The error was a context-length issue: the plugin sends a system prompt plus a long conversation history plus the user’s question, and Jan’s context window (the maximum amount of text the model can read at once) was truncating the conversation history at a point that broke the formatting. Ollama handles the same conversation length with the same model without the truncation, because Ollama’s default context window for Qwen 2.5 7B is 32K tokens (tokenization units, roughly 0.75 words each) and Jan’s default was 4K.
I tried to change the context window in Jan. The setting is in the model configuration, but the field does not accept values above 8192. The setting is documented as accepting up to 32K. The setting does not accept up to 32K. I dug through the GitHub issues, found a thread confirming the bug, found a pull request fixing the bug, and found that the fix had not been merged. The fix was on a branch I could compile from source, and I am not going to compile Jan from source on a Tuesday. I went back to Ollama.
This is the structural difference. Ollama is a tool that the user is expected to configure. Jan AI is a product that the user is expected to use. The product approach is the right one for the 90% of users who do not want to configure anything. The tool approach is the right one for the 10% of users who need to do something the product did not anticipate, and I am in that 10% for local AI the same way I am in that 10% for most developer tools.
What I would tell past me
If I could send a message back to the version of me that downloaded Jan AI last week, I would say three things.
- Local AI tools are not all the same. The product approach (Jan, LM Studio, GPT4All) is right for users who want a local ChatGPT and do not want to configure anything. The tool approach (Ollama, llama.cpp) is right for users who want to integrate, customize, and load custom models. Pick the one that matches what you are going to do with it.
- Context window is the most important number on the model card. A model that can only read 4K tokens is fine for one-shot questions. A model that can read 32K is fine for code review, long-document summarization, and the kind of multi-turn work that real workflows need. Check the context window before you check the parameter count.
- The model is the same either way. Qwen 2.5 7B running in Ollama and Qwen 2.5 7B running in Jan are the same model with the same weights and the same quality. The wrapper matters. The model is a commodity.
Trade-offs
Ollama is not free in time. The first time I set it up, I spent an afternoon getting the right CUDA version, configuring Open WebUI, and figuring out the Modelfile format. The system service that auto-starts Ollama on boot took me a half-day to get right on the home server, and I have had to redo it twice after kernel updates. For a user who is going to use local AI once a week, the half-day of setup is not a good trade for the marginal quality improvement over Jan AI.
Money is not really the cost here, because both are free at the model layer. The cost is in the optional GPU: I bought an NVIDIA RTX 3060 (a 12GB consumer GPU) for the home server, which was about $270, and that is the only reason local models run at a usable speed. A user running on integrated graphics (the basic GPU built into the CPU) is going to get 4-5 tokens per second on a 7B model, which is fast enough for short prompts and slow enough to be unusable for long answers.
Features are the third cost, and the one that is going to get better. Jan AI’s product approach means the UI is going to keep improving: built-in RAG (retrieval-augmented generation, a way to feed the model your own documents as context), built-in web search, and a polished mobile app are all on the Jan roadmap. Ollama’s tool approach means the user is going to keep needing to wire those features up themselves. For a user who is comfortable with the wiring, the trade is worth it. For a user who is not, the wiring is the product.
Upside is real too. I now run a local model for roughly 60% of the prompts I would have sent to ChatGPT, which is roughly $20 per month in ChatGPT Plus that I am not paying. The 14-day experiment was worth it because it confirmed that the local path is workable for the kind of writing and coding work I do, and the gap to ChatGPT is closing every quarter.