Ollama vs vLLM vs llama.cpp: Local LLMs

The week DeepSeek released its V4 Preview weights, my team's Slack turned into a runtime argument. One engineer had it running on a MacBook by dinner. Another spent two days wiring up a Blackwell node to serve it to the whole team. A third squeezed a quantized build onto a mini PC in the corner of the office. All three were "running DeepSeek locally," and all three were right, because they were using different tools for different jobs.

That's the thing nobody tells you about self-hosting open-weight models: the model is only half the decision. The runtime you wrap it in decides whether you get a 30-second toy demo or a production endpoint that holds up under load. As of July 2026, with the full DeepSeek V4 open-weight launch expected mid-month, three runtimes own the conversation: Ollama, vLLM, and llama.cpp. They are not competitors so much as tools with sharp, non-overlapping edges. Here's how to pick.

The three runtimes, in one paragraph each

Ollama is the "just works" option for local development. It wraps model download, quantization, and an OpenAI-compatible server behind a single command. Its 0.30 line, which opened with v0.30.0 on May 13, 2026, added a native MLX engine on Apple Silicon that leans on the unified-memory architecture, with the latest v0.30.10 shipping June 17. On M5-class chips it now taps the GPU Neural Accelerators for faster time-to-first-token. If you develop on a Mac, this is the path of least resistance.

vLLM is the throughput monster for GPU serving. It exists to answer one question: how many concurrent requests can I push through a GPU without the tail latency falling apart? Its answer is PagedAttention and continuous batching, and it is the runtime NVIDIA reaches for on new silicon. vLLM v0.21.0 stabilized DeepSeek V4 support on Blackwell, with early numbers near 3,500 tokens/sec per GPU on GB300-class hardware. This is not a desktop tool. It's what you put behind an internal API.

llama.cpp is the portable, quantized workhorse that runs almost anywhere: CPU-only servers, old GPUs, a Raspberry Pi, a gaming laptop. It's the C/C++ engine that half the ecosystem (Ollama included) is built on. In May 2026 it merged Multi-Token Prediction support to master (PR #22673, feature-flagged behind --mtp), a built-in speculative-decoding trick that roughly doubled single-user throughput on models like Qwen 3.6 27B. When portability and quantization matter more than raw concurrency, this is your runtime.

Side by side

The rows that matter most are "primary job" and "concurrency." Ollama and llama.cpp are optimized for one-to-a-few users; vLLM is optimized for many. Reach for the wrong one and you'll either wait ten minutes for a pip install you didn't need, or watch a desktop tool collapse the moment a second user shows up.

Pick this for that job

You're a developer prototyping on a laptop. Use Ollama. You want to test a prompt, wire up a quick RAG pipeline, or run a coding assistant offline without thinking about CUDA versions. One command and you have an endpoint:

# Pull and chat with a model locally
ollama run deepseek-v4-flash

# Or serve it on an OpenAI-compatible endpoint
ollama serve
# → POST http://localhost:11434/v1/chat/completions

You're serving a model to a team or product. Use vLLM. The moment you have real concurrency, throughput and tail latency become the whole game, and continuous batching is what keeps p99 sane under load:

# Serve a model with an OpenAI-compatible API
vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --tensor-parallel-size 2 \
  --max-model-len 131072

You need it to run anywhere, cheaply. Use llama.cpp. Edge boxes, CPU-only VMs, air-gapped environments, or a $500 mini PC doing overnight batch work. Grab a quantized GGUF, and with MTP you get a free speed bump on supported models:

# Run a quantized model with Multi-Token Prediction
llama-server -m deepseek-v4-flash-Q4_K_M.gguf \
  --mtp --host 0.0.0.0 --port 8080

My mild opinion: most teams over-reach for vLLM too early. If you have fewer than a handful of concurrent users, Ollama or a llama.cpp server will serve you fine at a fraction of the operational weight, and you can graduate to vLLM the day your traffic actually justifies a dedicated GPU. Serving infrastructure you don't need is just cloud cost with extra steps.

Where the DeepSeek V4 launch changes the math

DeepSeek's V4 family is what makes this comparison timely. The V4 Preview dropped April 24, 2026, in two shapes: V4 Pro (1.6T total parameters, 49B active) and V4 Flash (284B total, 13B active), both with a 1M-token context window. The Flash variant is the interesting one for self-hosting, because a 13B-active MoE punches far above its memory footprint.

That size profile maps cleanly onto the three runtimes. Flash quantizes down to something a serious workstation can hold, which is llama.cpp and Ollama territory. Pro, with its 1.6T total weights and long-context appetite, wants the memory bandwidth and batching of a multi-GPU vLLM deployment. Same model family, three completely different deployment stories, which is exactly why "run it locally" stopped being a single answer.

Whichever runtime you land on, the discipline around it is the same as any model in production: measure quality with proper evals, watch for hallucinations once real users arrive, and treat the endpoint like any other service you operate. For the broader picture, our DevOps and cloud category covers the deployment and cost side.

Frequently Asked Questions

Can I use the same GGUF model file across all three runtimes?

Ollama and llama.cpp both speak GGUF natively, so a quantized file often moves between them with little friction; Ollama is essentially a friendly wrapper over the same engine. vLLM is different, it's built around GPU-native formats like FP8 and safetensors rather than GGUF, so you generally serve the original or an FP8-quantized checkpoint there, not a Q4_K_M GGUF.

Do I need a GPU to run a local LLM in 2026?

No. llama.cpp runs entirely on CPU, and Ollama falls back to CPU when no GPU is present, though both are much faster with acceleration. A GPU becomes non-negotiable only when you move to vLLM for concurrent serving, where the whole point is saturating GPU memory bandwidth across many requests at once.

Which runtime is fastest?

It depends on the workload, and the honest answer defeats the question. For a single user, llama.cpp with MTP or Ollama's MLX engine is excellent and often feels instant. For many concurrent users, vLLM wins decisively because continuous batching keeps the GPU busy instead of idling between requests. Single-stream speed and aggregate throughput are different metrics; don't optimize the wrong one.

Is running models locally actually cheaper than an API?

Sometimes. If you have steady, high volume and already own the hardware, self-hosting can beat per-token API pricing, and it keeps sensitive data in-house. But for spiky or low volume, a hosted API is usually cheaper once you count GPU depreciation, electricity, and the engineer-hours to operate it. Run the numbers on your actual usage before committing, the same way you would for any cloud cost decision.