Ollama peaks at 41 tok/s. vLLM hits 793. TGI is in maintenance mode. Here's the self-hosting guide I wish existed before I started.
Get new posts delivered to your inbox. No spam, unsubscribe anytime.
Our OpenAI bill crossed $19,000 in February. The CTO asked me to "look into self-hosting." I spun up Ollama on my MacBook in 20 minutes, felt like a genius for 48 hours, then tried to serve 50 concurrent users with it and watched it collapse at 41 tokens per second. Three frameworks, two GPU providers, and one very expensive lesson later, I finally have a self-hosted stack that actually works. This is the guide I wish someone had written before I started.
Let me kill the hype upfront: self-hosting LLMs is not cheaper for most teams. Below 50 million tokens per day, APIs are almost always cheaper. The crossover from API to self-hosted happens at roughly 40-100 million tokens per month depending on the model and instance type.
The math works like this. Groq charges $0.11 per million tokens for Llama 4 Scout. A self-hosted Llama model on an H100 costs about $1.49-2.99/hour on cloud GPUs, which translates to $1,080-2,150 per month running 24/7. Add 20-40 hours of engineer time for initial setup and 5-10 hours per month for ongoing maintenance at $75-150/hour fully loaded cost.
So why self-host at all? Three reasons:
If none of these apply to you, stop reading and keep using the API. I'm serious. Self-hosting is operationally complex and the cost savings only work at scale.
Here's the mistake every comparison article makes: they benchmark vLLM, TGI, and Ollama on the same workload and declare a winner. That's like benchmarking a Formula 1 car, a delivery truck, and a bicycle on a racetrack. They're not competing — they're solving different problems.
Ollama hit 52 million monthly downloads in Q1 2026, with over 90,000 GitHub stars. It's the Docker of LLMs — one command pulls and runs a model:
# That's it. You're running Llama 3.2 locally.
ollama run llama3.2
# Or serve it as an API
ollama serve
curl http://localhost:11434/api/generate -d '{
"model": "llama3.2",
"prompt": "Explain PagedAttention in one paragraph"
}'
Ollama wraps llama.cpp under the hood, handles quantization automatically, exposes an OpenAI-compatible API, and supports NVIDIA (CUDA), Apple Silicon (Metal), and AMD (ROCm) GPUs. The model library includes Llama, Mistral, Gemma, Phi, Qwen, and 135,000+ GGUF-formatted models on HuggingFace.
But here's what nobody tells you: Ollama caps at roughly 4 parallel requests by default and peaks around 41 tokens per second under load. At 10 concurrent users, total throughput is only ~150 tokens per second. It has no PagedAttention, no continuous batching, and no multi-node support.
Ollama is perfect for: local development, prototyping, demos, single-user tools, CLI assistants, and small internal apps with under 5 concurrent users. It is not for production serving at scale.
vLLM is what you deploy when performance matters. Built at UC Berkeley, its core innovation is PagedAttention — inspired by virtual memory paging in operating systems, it breaks the KV cache into fixed-size blocks that can be stored anywhere in GPU memory and reused across requests.
The impact is massive. Existing LLM serving systems waste 60-80% of KV cache memory. vLLM reduces that waste to under 4%. That memory efficiency translates directly to throughput: vLLM achieves up to 24x higher throughput than HuggingFace Transformers and hits 793 tokens per second where Ollama manages 41.
# Install and serve
pip install vllm
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--tensor-parallel-size 1 \
--max-model-len 8192 \
--gpu-memory-utilization 0.90
# OpenAI-compatible API out of the box
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [{"role": "user", "content": "Hello"}]
}'
vLLM V1 (January 2025) brought a major architectural upgrade with 1.7x speedup, and the latest releases add full NVIDIA Blackwell SM120 and H200 support. Key features:
vLLM is the default choice for production LLM serving in 2026. But it requires a GPU with sufficient VRAM, more setup than Ollama, and operational knowledge of CUDA, model loading, and memory management.
This is the part most comparison articles get wrong in 2026, because they're still recommending TGI as a production option.
Hugging Face put TGI into maintenance mode in December 2025. Only minor bug fixes and documentation improvements are accepted. Hugging Face is now contributing directly to the vLLM project and integrating vLLM as a TGI backend. They're also partnering with the llama.cpp team for CPU-based inference.
TGI was a great framework. It introduced continuous batching and Flash Attention to the ecosystem. In 2024, TGI v3.0 was 13x faster than vLLM on long prompts. But the ecosystem moved, and Hugging Face made the pragmatic decision to converge rather than maintain two separate engines.
My recommendation: Don't start new deployments on TGI. If you're already running TGI in production, plan a migration to vLLM or SGLang within the next 6-12 months. The clock is ticking.
Most comparison articles show single-metric benchmarks. Here's the full picture, based on Llama 3.1 8B on a single GPU:
| Metric | Ollama | TGI | vLLM |
|---|---|---|---|
| Single-user throughput | 65 tok/s | 110 tok/s | 140 tok/s |
| 10 concurrent users | ~150 tok/s total | ~500 tok/s total | ~800 tok/s total |
| Peak throughput | 41 tok/s per user | ~50 tok/s per user | 793 tok/s total |
| P99 latency (peak) | 673 ms | ~200 ms | 80 ms |
| GPU utilization | 40-60% | 65-80% | 85-92% |
| PagedAttention | No | Yes (via vLLM kernels) | Yes (native) |
| Continuous batching | No | Yes | Yes |
| Multi-GPU support | Limited | Yes | Yes (tensor parallelism) |
| Setup time | 5 minutes | 30 minutes | 45 minutes |
| OpenAI-compatible API | Yes | Yes | Yes |
| Structured output | Yes | Limited | Yes |
| Maintenance status | Active | Maintenance mode | Active |
The gap between Ollama and vLLM isn't 2x. It's 19x at peak throughput. If you're serving more than a handful of concurrent users, that difference isn't a nice-to-have — it's the difference between a responsive application and a queue that backs up indefinitely.
I'd be doing you a disservice if I didn't mention SGLang. It's the fastest inference engine in 2026 for multi-turn workloads.
SGLang delivers approximately 16,200 tokens per second on H100s, compared to vLLM's ~12,500 — a 29% throughput advantage. Its core innovation is RadixAttention: it stores cached prefixes in a radix tree, enabling up to 5x faster inference for workloads with shared prefixes.
SGLang already powers over 400,000 GPUs across xAI (Grok 3), Microsoft Azure, LinkedIn, and Cursor. If your workload involves multi-turn conversations, shared document context, or structured output, SGLang is worth evaluating alongside vLLM.
But vLLM has the more mature ecosystem, larger community, and broader model support. For most teams in 2026, vLLM is still the safer default.
You don't need an $30,000 H100 to run serious models. Quantization compresses model weights from 16-bit floats to 4-bit or 5-bit integers, cutting memory requirements by 70-75% with 92-95% quality retention.
The practical cheat sheet:
| Quantization | Quality Retention | VRAM for 7B | VRAM for 70B | Best For |
|---|---|---|---|---|
| FP16 (no quant) | 100% | ~14 GB | ~140 GB | Benchmarking only |
| Q8_0 | ~99% | ~8 GB | ~75 GB | Quality-critical production |
| Q5_K_M | ~97% | ~5.5 GB | ~50 GB | Production sweet spot |
| Q4_K_M | ~95% | ~4.5 GB | ~40 GB | Most users, best balance |
| Q3_K_M | ~90% | ~3.5 GB | ~32 GB | Extremely constrained |
Q4_K_M is the mainstream choice — good for most tasks with acceptable quality loss. Q5_K_M is recommended for critical applications where you can afford the extra VRAM.
For Ollama, quantization is automatic — ollama run llama3.2 pulls a Q4 variant by default. For vLLM, you'll typically use AWQ or GPTQ quantized models from HuggingFace:
# vLLM with AWQ quantization
vllm serve TheBloke/Llama-2-70B-Chat-AWQ \
--quantization awq \
--tensor-parallel-size 2 \
--max-model-len 4096
The fundamental constraint for LLM serving is VRAM, not compute FLOPS. The formula: model parameters x bytes per parameter (set by quantization) + KV cache overhead + 2-4 GB framework runtime.
For development (Ollama):
For production (vLLM):
Current cloud GPU pricing for a single card running 24/7:
| GPU | Cloud Price/Hr | Monthly Cost | VRAM | Best For |
|---|---|---|---|---|
| A100 40GB | ~$1.19 | ~$860 | 40 GB | Budget production |
| A100 80GB | ~$1.59 | ~$1,145 | 80 GB | Standard production |
| H100 80GB | ~$1.49-2.99 | ~$1,080-2,150 | 80 GB | High-throughput production |
| RTX 4090 | Buy: ~$1,600 | Electricity only | 24 GB | On-prem development |
Here's the deployment path I'd follow today if starting from scratch.
Phase 1: Prototype with Ollama (Day 1)
# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
# Pull and test your target model
ollama pull llama3.1:8b
ollama run llama3.1:8b
# Test the API
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "llama3.1:8b", "messages": [{"role": "user", "content": "Hello"}]}'
Build your application against Ollama's OpenAI-compatible API. All your code will work unchanged when you switch to vLLM later — same API format.
Phase 2: Benchmark your workload (Days 2-3)
Before committing to a GPU and framework, measure your actual traffic:
If the answers are "under 5 concurrent users" and "latency doesn't matter much" — stay on Ollama. You're done.
Phase 3: Deploy vLLM for production (Days 4-7)
# Docker deployment (recommended for production)
docker run --runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 8192 \
--gpu-memory-utilization 0.90
For multi-GPU setups:
# Tensor parallel across 2 GPUs for 70B models
docker run --runtime nvidia --gpus all \
-p 8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 2 \
--max-model-len 4096 \
--gpu-memory-utilization 0.90
Phase 4: Add monitoring and failover (Week 2)
Put a reverse proxy (Nginx, Caddy) in front with authentication. Never expose vLLM directly to the internet. Add health checks, log request latency, and set up alerts for GPU memory and throughput degradation.
Use LiteLLM as your proxy layer for cost tracking, rate limiting, and API key management. It adds load balancing across multiple vLLM instances and automatic failover to cloud APIs when your self-hosted cluster is overloaded.
Mistake 1: Using vLLM for development. I spent an entire day configuring CUDA, downloading model shards, and debugging tensor parallel settings — for a prototype that only I was using. Should've used Ollama for development and switched to vLLM only when deploying to production. The setup overhead is zero with Ollama.
Mistake 2: Undersizing RAM. Our first production deployment had 32 GB system RAM for a 70B quantized model (~35 GB at Q4). The model loaded fine but the system became unusable — no headroom for the OS, KV cache overflow to CPU memory, constant swapping. System RAM should be at least 1.5x model size. We upgraded to 64 GB and the problems vanished.
Mistake 3: Ignoring model quality on our actual use case. We picked the model with the best MMLU benchmark score. It was terrible at our specific task (customer support ticket classification). Benchmark performance doesn't match real-world task quality. We wasted two weeks before realizing a smaller, fine-tuned model outperformed the larger one on our data.
Mistake 4: No fallback to cloud APIs. GPU hardware fails. CUDA OOMs happen. When our self-hosted endpoint went down at 2 AM, our entire product was down. Now we route through LiteLLM with automatic fallback to OpenAI when self-hosted latency exceeds 5 seconds. The cloud API bill during outages is negligible compared to the cost of downtime.
Mistake 5: Running at 100% GPU utilization. Sounds efficient. Isn't. At 100% GPU memory utilization, there's no headroom for KV cache growth during longer conversations. We had random OOMs during peak hours because a single long-context request would push memory over the edge. We settled on 90% GPU memory utilization (--gpu-memory-utilization 0.90) — the remaining 10% acts as a buffer.
Stop googling benchmarks. Answer these four questions:
How many concurrent users?
What's your budget?
How sensitive is your data?
How much ops capacity do you have?
The self-hosting discourse is full of people who either think everyone should self-host (they're selling GPU compute) or nobody should (they're selling API access). The truth is boring: it depends on your scale, your data sensitivity, and your willingness to maintain infrastructure.
I think Ollama is one of the best developer tools released in the last three years. The fact that you can ollama run llama3.2 and have a working LLM in 20 seconds is extraordinary. But I also think too many people try to run Ollama in production and wonder why it falls over. It's a development tool. Use it for development.
I think vLLM is the correct default for production self-hosting in 2026. Not because it's always the fastest — SGLang beats it on multi-turn by 29% — but because it has the largest community, the most integrations, and the deepest documentation. When you hit a problem at 3 AM, there are Stack Overflow answers for vLLM. There aren't for SGLang.
I think TGI is effectively dead for new deployments. Hugging Face made the right call putting it in maintenance mode — maintaining two inference engines makes no sense when you can contribute to the one that's winning. If you're on TGI today, start planning your migration.
I think most teams that self-host will save less money than they expect and spend more time on operations than they budgeted. The break-even point is real, but it's higher than the blog posts claim once you factor in engineer time, GPU redundancy, and the operational overhead of keeping a CUDA stack healthy. Self-host because you need to — for data privacy, latency, or genuine volume economics — not because it sounds cool.
And I think the biggest unlock isn't the framework choice. It's quantization. The fact that you can run a 70B parameter model on a single $1,600 RTX 4090 with Q4 quantization and get 95% of the original quality is the real revolution. Three years ago, that model required a $200,000 cluster. Now it runs on hardware you can buy at Best Buy.
Start with Ollama. Build your app. Measure your traffic. And only when the numbers actually justify it, deploy vLLM on a rented GPU. Everything else is premature optimization.