A practical guide to fine-tuning LLMs with LoRA, QLoRA, Unsloth, and OpenAI. Real costs, real code, and when to fine-tune vs RAG.
Get new posts delivered to your inbox. No spam, unsubscribe anytime.
Last month I spent $14 fine-tuning a 7B parameter model on 8,000 customer support conversations. The model went from giving generic, vaguely helpful answers to sounding exactly like our best support agent — tone, terminology, formatting, everything. A colleague had spent three weeks engineering prompts to get the same behavior out of GPT-4o. His version was still worse.
That's the thing about fine-tuning nobody tells you upfront: it's not about making the model smarter. It's about making the model behave differently. And once you understand that distinction, a lot of the confusion around when to fine-tune (and when not to) just evaporates.
Before you fine-tune anything, you need to be honest about what problem you're actually solving. I've watched teams burn weeks fine-tuning models when they should have been building a RAG pipeline. And I've seen teams build elaborate retrieval systems when a 20-line system prompt would have done the job.
Here's the mental model I use:
Or, as IBM's 2026 best practices guide puts it: "Put volatile knowledge in retrieval, put stable behavior in fine-tuning."
That framing has saved me from bad decisions multiple times. If your data changes weekly — product catalogs, pricing, documentation — that's retrieval. If you want the model to always respond in a specific format, always use certain terminology, or always match a particular tone — that's fine-tuning.
| Approach | Best For | Cost | Effort | Latency |
|---|---|---|---|---|
| Prompt Engineering | Quick fixes, prototypes | Free | Minutes | Same |
| RAG | Dynamic knowledge, documents | Low-Medium | Days | +100-300ms |
| Fine-Tuning | Behavior, tone, format, domain style | Medium | Days-Weeks | Same or faster |
| Fine-Tuning + RAG | Domain behavior + live knowledge | Higher | Weeks | +100-300ms |
There's also a practical threshold people don't talk about enough. Once your system prompt plus few-shot examples start pushing past a significant chunk of your context window, you're paying for those tokens on every single request. Fine-tuning bakes that behavior into the weights. Your inference prompts get shorter. Your per-request costs drop. Your latency drops. With models now supporting around 200K token context windows, you have more room — but stuffing 50K tokens of examples into every API call is still wasteful.
I need to kill a misconception here. Fine-tuning does not teach the model new facts. If your base model doesn't know that your company's return policy is 30 days, fine-tuning won't reliably teach it that. The model might memorize the fact from your training data, but it might also hallucinate a different number next Tuesday.
What fine-tuning does change:
What it doesn't change well:
The RAG market is projected to hit $47 billion by 2034, which tells you something: most companies need both approaches. Fine-tune for behavior, retrieve for knowledge. That combination is what actually works in production.
Here's where things have changed dramatically in the past year. Fine-tuning used to be expensive enough to make you think twice. Now? Not so much.
OpenAI slashed prices hard with GPT-4.1:
| Model | Training Cost (per 1M tokens) | Inference Input | Inference Output |
|---|---|---|---|
| GPT-4o | $25.00 | $2.50 | $10.00 |
| GPT-4.1 | $3.00 | $2.00 | $8.00 |
| GPT-4.1-mini | $0.80 | $0.40 | $1.60 |
That's an 88% price drop from GPT-4o to GPT-4.1 for training. A dataset of 10,000 training examples averaging 500 tokens each is 5 million tokens. At GPT-4.1 rates, that's $15. Fifteen dollars to fine-tune a frontier model on your data.
GPT-4.1-mini is even more absurd — that same dataset costs $4.
Here's a minimal fine-tuning job using OpenAI's API:
from openai import OpenAI
client = OpenAI()
# Upload your training file (JSONL format)
training_file = client.files.create(
file=open("training_data.jsonl", "rb"),
purpose="fine-tune"
)
# Start the fine-tuning job
job = client.fine_tuning.jobs.create(
training_file=training_file.id,
model="gpt-4.1-mini",
hyperparameters={
"n_epochs": 3,
"batch_size": "auto",
"learning_rate_multiplier": "auto"
}
)
print(f"Job started: {job.id}")
print(f"Status: {job.status}")
Your training data needs to be JSONL with this format:
{"messages": [{"role": "system", "content": "You are a support agent for Acme Corp."}, {"role": "user", "content": "How do I reset my password?"}, {"role": "assistant", "content": "Head to acme.com/reset, enter your email, and click the link we send you. Takes about 30 seconds."}]}
{"messages": [{"role": "system", "content": "You are a support agent for Acme Corp."}, {"role": "user", "content": "My order hasn't arrived"}, {"role": "assistant", "content": "I can look into that. What's your order number? I'll check the tracking status right now."}]}
Each line is one training example. You want at least 50-100 examples for basic behavior changes, and 1,000-10,000 for more complex domain adaptation. More data generally helps, but with diminishing returns past about 10K examples for most use cases.
If you want to fine-tune open-source models — Llama 3, Mistral, Phi-3 — or if you need to keep your data on-premise, LoRA (Low-Rank Adaptation) is the technique that changed everything.
The problem with full fine-tuning: you're updating every parameter in the model. A full fine-tune of a 7B parameter model requires roughly $50,000 in GPU hardware. Multiple A100s or H100s, days of training time, and enough VRAM to hold the entire model plus optimizer states.
LoRA's insight is simple: instead of updating all parameters, freeze the original weights and train small "adapter" matrices that modify the model's behavior. These adapters are typically less than 1% of the original model size. You get most of the benefit at a fraction of the cost.
QLoRA takes this further by quantizing the base model to 4-bit precision. The same 7B model that needed $50,000 in GPUs for full fine-tuning? QLoRA fits it on a single 16GB GPU like an RTX 4090, costing about $1,500. If you rent cloud compute, a QLoRA fine-tune on an H100 runs $10-16 for 8-12 hours.
And the quality trade-off? QLoRA achieves 80-90% of full fine-tuning quality for most tasks. That last 10-20% matters if you're pushing state-of-the-art benchmarks. It doesn't matter if you're making a customer support bot sound right.
Here's what a LoRA config looks like in practice:
from peft import LoraConfig, get_peft_model, TaskType
lora_config = LoraConfig(
r=16, # rank of the adapter matrices
lora_alpha=32, # scaling factor
target_modules=[
"q_proj", "k_proj", # attention query and key
"v_proj", "o_proj", # attention value and output
"gate_proj", "up_proj", # MLP layers
"down_proj"
],
lora_dropout=0.05,
bias="none",
task_type=TaskType.CAUSAL_LM
)
model = get_peft_model(base_model, lora_config)
# Check trainable parameters
model.print_trainable_parameters()
# Output: trainable params: 13,631,488 || all params: 7,241,732,096 || trainable%: 0.188%
The r=16 is the rank — it controls how expressive your adapters are. Higher rank = more parameters = more capacity to learn, but also more memory and overfitting risk. I've found r=16 is the sweet spot for most tasks. Bump to 32 or 64 if you have a lot of training data and the model isn't fitting well enough.
The target_modules list tells LoRA which layers to adapt. For most Llama-family models, targeting all the attention projections plus the MLP layers gives you the best results. Some people only target the attention layers to save memory, but I've seen noticeably better results when you include the MLP.
If you're fine-tuning open-source models in 2025-2026, you should know about Unsloth. It's a library that wraps Hugging Face Transformers and PEFT with aggressive optimizations. The numbers: 2x faster training, 60% less memory usage, zero accuracy loss. Those aren't marketing claims — I've verified them on my own runs.
Here's a complete fine-tuning script using Unsloth:
from unsloth import FastLanguageModel
from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset
# Load model with 4-bit quantization
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/Meta-Llama-3.1-8B-bnb-4bit",
max_seq_length=2048,
load_in_4bit=True,
)
# Add LoRA adapters
model = FastLanguageModel.get_peft_model(
model,
r=16,
lora_alpha=16,
lora_dropout=0,
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"
],
use_gradient_checkpointing="unsloth",
)
# Format your training data
alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction:
{}
### Input:
{}
### Response:
{}"""
def formatting_func(examples):
texts = []
for instruction, inp, output in zip(
examples["instruction"],
examples["input"],
examples["output"]
):
text = alpaca_prompt.format(instruction, inp, output)
texts.append(text)
return {"text": texts}
# Load and format dataset
dataset = load_dataset("json", data_files="my_training_data.json", split="train")
dataset = dataset.map(formatting_func, batched=True)
# Train
trainer = SFTTrainer(
model=model,
tokenizer=tokenizer,
train_dataset=dataset,
dataset_text_field="text",
max_seq_length=2048,
args=TrainingArguments(
per_device_train_batch_size=2,
gradient_accumulation_steps=4,
warmup_steps=5,
num_train_epochs=3,
learning_rate=2e-4,
fp16=True,
logging_steps=10,
output_dir="outputs",
optim="adamw_8bit",
seed=42,
),
)
trainer.train()
# Save the LoRA adapter
model.save_pretrained("my-fine-tuned-model")
tokenizer.save_pretrained("my-fine-tuned-model")
# Or merge and save as full model for deployment
model.save_pretrained_merged(
"my-fine-tuned-model-merged",
tokenizer,
save_method="merged_16bit",
)
A few things I want to call out:
load_in_4bit=True — this is QLoRA. The base model gets quantized to 4-bit, which is why a 7-8B model fits on 16GB of VRAM. Without this, you need 32+ GB.
use_gradient_checkpointing="unsloth" — Unsloth's custom gradient checkpointing implementation. It trades a bit of compute for a lot of memory savings. Always turn this on.
optim="adamw_8bit" — 8-bit Adam optimizer. The optimizer states are one of the biggest memory hogs in training. Using 8-bit precision here cuts memory usage significantly with minimal impact on convergence.
gradient_accumulation_steps=4 — with a batch size of 2 and 4 accumulation steps, your effective batch size is 8. This lets you simulate larger batches without needing more VRAM.
The whole script runs on a single RTX 4090 or an equivalent cloud GPU. On RunPod, that's about $1-2/hour. A typical fine-tuning run on 5,000-10,000 examples takes 2-4 hours. So you're looking at $2-8 total.
I'm going to be blunt: data quality matters more than anything else in fine-tuning. More than the model choice. More than the hyperparameters. More than the training framework. If your training data is bad, your fine-tuned model will be confidently, consistently bad.
Here's what good training data looks like:
Consistent format. Every example should follow the same structure. If you want the model to output JSON, every training example should have JSON output. If you want concise answers, every example should have concise answers. The model learns patterns. Inconsistent patterns teach the model to be inconsistent.
Representative distribution. If 80% of your real queries are about billing, 80% of your training data should be about billing. I've seen teams fine-tune on a curated dataset that overrepresented edge cases, then wonder why the model was terrible at common questions.
At least 100 examples. OpenAI recommends 50-100 as a minimum and I agree. You can see improvements with as few as 50, but the results get meaningfully better at 500-1,000. Past 10,000 you're usually in diminishing returns territory unless your task is very complex.
Clean, correct outputs. Every response in your training set should be exactly what you want the model to produce. Not approximately. Exactly. If you wouldn't want the model to say it, don't put it in the training data. Go through your data line by line if you have to. I know it's tedious. Do it anyway.
Here's a Python script to validate and prepare your JSONL training data:
import json
import tiktoken
def validate_training_data(filepath: str) -> dict:
"""Validate JSONL training file for OpenAI fine-tuning."""
encoding = tiktoken.encoding_for_model("gpt-4o")
errors = []
total_tokens = 0
examples = []
with open(filepath, "r") as f:
for i, line in enumerate(f, 1):
try:
data = json.loads(line)
except json.JSONDecodeError:
errors.append(f"Line {i}: Invalid JSON")
continue
if "messages" not in data:
errors.append(f"Line {i}: Missing 'messages' key")
continue
messages = data["messages"]
# Check roles
roles = [m["role"] for m in messages]
if roles[-1] != "assistant":
errors.append(f"Line {i}: Last message must be from assistant")
if "user" not in roles:
errors.append(f"Line {i}: Must contain at least one user message")
# Count tokens
text = " ".join(m["content"] for m in messages)
tokens = len(encoding.encode(text))
total_tokens += tokens
examples.append({"line": i, "tokens": tokens})
# Cost estimates
cost_4_1 = (total_tokens / 1_000_000) * 3.00
cost_4_1_mini = (total_tokens / 1_000_000) * 0.80
return {
"total_examples": len(examples),
"total_tokens": total_tokens,
"avg_tokens_per_example": total_tokens // max(len(examples), 1),
"estimated_cost_gpt4_1": f"${cost_4_1:.2f}",
"estimated_cost_gpt4_1_mini": f"${cost_4_1_mini:.2f}",
"errors": errors,
}
result = validate_training_data("training_data.jsonl")
print(json.dumps(result, indent=2))
Run this before you submit any fine-tuning job. It catches the obvious mistakes — malformed JSON, missing roles, wrong message ordering — and tells you exactly what it'll cost. I've added this to every fine-tuning pipeline I build now.
I've built this flowchart from experience. It's not perfect, but it'll save you from the most common bad decisions.
Step 1: Can you solve it with a better prompt? Seriously. Before you fine-tune, spend a day writing a really good system prompt with 3-5 few-shot examples. If that gets you to 80% of where you need to be, you might not need fine-tuning at all. Prompt engineering is free and immediate.
Step 2: Is the problem knowledge or behavior? If the model doesn't know something (your product specs, your internal docs, recent events), that's a knowledge problem. Use RAG. If the model knows enough but doesn't respond the way you want (wrong format, wrong tone, too verbose, too cautious), that's a behavior problem. Fine-tune.
Step 3: Do you have good training data? If you have fewer than 50 high-quality examples of the behavior you want, you probably don't have enough signal for fine-tuning to help. Collect more data first. Fine-tuning on bad data makes things worse, not better.
Step 4: Is the behavior stable? If what you want the model to do will change next month, fine-tuning is a bad fit. Every change means a new fine-tuning run. For volatile requirements, keep the flexibility of prompt engineering or RAG.
Step 5: Pick your path.
| Situation | Recommendation | Estimated Cost |
|---|---|---|
| Small team, proprietary data, need privacy | QLoRA on open-source model, self-hosted | $2-16 (cloud GPU) |
| Startup, fast iteration, data can go to OpenAI | OpenAI fine-tuning API (GPT-4.1-mini) | $4-50 |
| Enterprise, complex domain, large dataset | QLoRA on 70B model or OpenAI GPT-4.1 | $50-500 |
| Need best quality, budget is flexible | Full LoRA on 70B model with H100s | $500-5,000 |
Fine-tuning when you need RAG. The most common mistake. Someone wants the model to answer questions about their docs, so they fine-tune on Q&A pairs from those docs. It works — until the docs change. Then you need to re-fine-tune. RAG handles this naturally because the retrieval layer always sees the latest documents.
Not evaluating properly. You fine-tuned the model, it "feels" better on a few test inputs. Is it actually better? You need a held-out evaluation set — 50-100 examples the model never saw during training. Score them. Compare to the base model. Compare to the prompt-engineered version. Without numbers, you're guessing.
Overfitting on small datasets. If you fine-tune on 100 examples for 10 epochs, the model will memorize those examples. It'll perform perfectly on your training data and terribly on everything else. Watch your training loss. If it's dropping to near zero, you're overfitting. Reduce epochs, increase data, or add regularization (LoRA dropout helps).
Ignoring the base model's strengths. The base model already knows a lot. Your fine-tuning data should teach new behavior, not reiterate things the model already does well. If GPT-4.1 already formats JSON correctly, you don't need 5,000 examples of JSON formatting. Focus your data on what's actually different about your use case.
Training on synthetic data without filtering. Using GPT-4 to generate training data for a smaller model is a legitimate strategy. But you have to filter the outputs. I generate 2-3x the data I need, score each example against my quality criteria, and only keep the top tier. Unfiltered synthetic data includes all of GPT-4's bad habits along with the good ones.
Here's my honest take after fine-tuning maybe a dozen models over the past year:
Fine-tuning is the most underused technique in applied AI right now. Most teams default to either prompt engineering (easy but limited) or RAG (good for knowledge but doesn't change behavior). Fine-tuning occupies this middle ground that people skip because they think it's hard or expensive. It's neither. Not anymore.
With QLoRA and Unsloth, you can fine-tune a 7B model on your laptop for the cost of lunch. With OpenAI's API, you can fine-tune GPT-4.1-mini for less than the price of a coffee. The tooling has gotten ridiculously good.
But here's the nuance: fine-tuning is only as good as your data. I've seen teams spend weeks on training infrastructure and hyperparameter tuning when they should have spent that time curating better training examples. The model doesn't care about your learning rate schedule if your training data is inconsistent garbage. Get the data right. Everything else follows.
The combination that actually works in production is fine-tuning plus RAG. Fine-tune for behavior — how the model talks, what format it uses, what it refuses to do. Use RAG for knowledge — the facts, the documents, the stuff that changes. Trying to do both with just one technique is where people get into trouble.
One more thing. If you're choosing between OpenAI fine-tuning and open-source fine-tuning, be honest about your constraints. OpenAI is easier, faster, and the results are often better because their base models are stronger. Open-source gives you control, privacy, and no per-token inference costs. Both are valid. Pick based on your actual requirements, not ideology.
The market is maturing fast. Fine-tuning APIs are getting cheaper. Open-source tooling is getting better. A year from now, fine-tuning a model on your own data will be as routine as training a classifier. The teams that figure this out early have a real advantage. The ones that keep stuffing 50 few-shot examples into every prompt are leaving performance and money on the table.