Fine-Tuning vs RAG vs Prompting: The Decision Tree

TL;DR: Fine-tuning, RAG, and prompting are not three options to pick from — they are three different layers. Prompting controls how the model says things, RAG controls what it does not know, and fine-tuning controls what it does by default. Prompt caching breaks even at just 2 requests inside a 5-minute TTL window, which is roughly 24 requests per hour. LoRA on an 8B model with the mainstream 2026 recipe (all 7 projections, r=16) compresses trainable parameters by 191.5x, not the 10000x from the paper abstract. And OpenAI has scheduled its self-serve fine-tuning platform to stop accepting new jobs for everyone on January 6, 2027. Every price and multiplier here comes from vendor documentation, verified as of October 14, 2026. This article contains no affiliate links.

Three layers, not three options

Almost every "should we fine-tune?" argument starts in the wrong place, because it treats three different layers as one button.

Here is the one-line test: if the model does not know, use RAG. If it knows but will not comply, use prompting, and only reach for fine-tuning when prompting stops holding.

That split is not folklore. The original RAG paper (arXiv:2005.11401, NeurIPS 2020, Lewis et al.) separates parametric memory — knowledge baked into weights — from non-parametric memory, a dense vector index over Wikipedia, and names two open problems for parametric models: providing provenance for their decisions, and updating their world knowledge. In other words, injecting knowledge was never fine-tuning's job. It is the reason RAG exists.

The six-step decision tree

Step 1: Is it missing knowledge or missing behavior?

Let the model answer once and look at the failure mode. Wrong facts means missing knowledge. Right facts but wrong format, tone, or length means missing behavior. Missing knowledge goes straight to RAG. Stop tuning the prompt.

Step 2: Does the knowledge change?

If it changes — price lists, inventory, internal docs, news — RAG is the only real answer. Burning volatile facts into weights creates a retraining debt you will pay forever.

Step 3: How many clean labeled examples do you have?

Fewer than 50, stay with prompting. OpenAI's own guidance is to start text SFT from 50 well-crafted examples with a hard floor of 10. That is a floor, not a promise that 50 is enough.

Step 4: Is your volume high enough to amortize anything?

This is the step everyone skips. The next section puts a number on it.

Step 5: Do you need provenance?

If you have to answer "which document did that sentence come from," you need RAG.

Step 6: Only now consider fine-tuning.

And first confirm the platform still lets you. See the shutdown schedule below.

Cost ledger 1: where prompt caching breaks even

Anthropic's official pricing documentation is explicit about the multipliers:

• 5-minute TTL cache write = 1.25x the base input price
• 1-hour TTL cache write = 2x the base input price
• Cache read (a hit) = 0.1x the base input price (0.05x on Claude Opus 5.5 and Sonnet 5.5)

The docs state the conclusion directly: caching pays off after one cache read for the 5-minute duration (1.25x write), or after two cache reads for the 1-hour duration (2x write). I ran the arithmetic from those multipliers and landed on exactly the same place:

The 1-hour TTL row needs 3 requests before it wins (2.20x against 3.00x), which also matches the two-cache-reads statement.

Turned into a threshold you can check against your own traffic: an average of at least 2 requests per 5-minute window, about 24 per hour, and caching already saves money. Below that, you never earn back the 1.25x write premium.

One more thing worth dating: Anthropic's API release note for October 7, 2026 cut the cache read price on Claude Sonnet 5.5 from $0.20 to $0.10 per million tokens, moving it from 0.1x to 0.05x. The same code got cheaper inside a week. That is why every number in this article carries a date.

Cost ledger 2: LoRA's real compression ratio, measured locally

Nearly every LoRA article quotes the abstract's line about reducing trainable parameters by 10,000 times and GPU memory requirements by 3 times (arXiv:2106.09685, Hu et al., submitted June 17, 2021). It is true. It is also true only for the paper's own experimental setup, not for the recipe you are running.

I ran the number locally with peft 0.21.2, transformers, and torch 2.14.1+cpu. The approach: build a graph from the public Llama 3 8B architecture (hidden 4096, intermediate 14336, 32 layers, 32 heads, 8 KV heads) on the meta device so nothing is actually allocated, then let peft inject the adapter and count trainable parameters for real:

import torch
from transformers import LlamaConfig, LlamaForCausalLM
from peft import LoraConfig, get_peft_model

DIM, FFN, LAYERS, HEADS, KV, VOCAB = 4096, 14336, 32, 32, 8, 128256

with torch.device("meta"):          # build the graph only, allocate nothing
    cfg = LlamaConfig(hidden_size=DIM, intermediate_size=FFN,
                      num_hidden_layers=LAYERS, num_attention_heads=HEADS,
                      num_key_value_heads=KV, vocab_size=VOCAB, rope_theta=500000)
    model = LlamaForCausalLM(cfg)

total = sum(p.numel() for p in model.parameters())          # 8,030,261,248

for r, mods in [(16, ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj"]),
                (8,  ["q_proj", "v_proj"])]:
    lcfg = LoraConfig(r=r, lora_alpha=2 * r, lora_dropout=0.0,
                      target_modules=mods, task_type="CAUSAL_LM")
    pm = get_peft_model(model, lcfg)
    tr = sum(p.numel() for p in pm.parameters() if p.requires_grad)
    print(f"r={r} modules={len(mods)} trainable={tr:,} share={tr/total*100:.3f}% compression={total/tr:.1f}x")
    pm = pm.unload()

Actual output:

r=16 modules=7  peft reports trainable = 41,943,040  share 0.522%  compression 191.5x
r=8  modules=2  peft reports trainable = 3,407,872   share 0.042%  compression 2356.4x

So the conclusion is blunt: with the mainstream 2026 recipe — q/k/v/o plus gate/up/down, seven projections, r=16, alpha = 2r — the compression ratio is 191.5x, not 10000x. Where does 10000x come from? I ran the paper's setup too: GPT-3 175B, q and v only, r=4, 96 layers gives 18,874,368 trainable parameters and 9,272x compression. That matches the claim.

Three things moved at once: the base shrank from 175B to 8B, the rank went from 4 to 16, and the targeted modules went from 2 to 7.

The memory ledger needs the same treatment. Using the usual mixed-precision AdamW accounting (bf16 weights 2 bytes, bf16 gradients 2 bytes, fp32 master 4 bytes, first moment 4 bytes, second moment 4 bytes — 16 bytes per parameter):

• Full fine-tuning of 8B: about 119.7 GiB of training state, which does not fit on a single 80GB card
• LoRA r=16 across all 7 modules: 41,943,040 trainable parameters, roughly 40 MiB of optimizer state

That is LoRA's actual value: not the pretty "10000x fewer parameters" line, but dropping the training barrier from a multi-GPU cluster to a single card. And do not forget the abstract's note that, unlike adapters, it adds no inference latency — LoRA weights can be merged back into the base.

Cost ledger 3: four discounts stack on a fine-tuned model's unit price

Most people budget fine-tuning as training cost only. But OpenAI's official pricing page (the Finetuning section, o4-mini-2025-04-16) lists several tiers, and the same fine-tuned model's input price spans 16x:

Input moves from $4.00 to $0.25, a factor of 16; output moves from $16.00 to $4.00, a factor of 4. Note that the $0.25 line is the cached-input tier — a fine-tuned model still benefits from prompt caching. The two are not in conflict.

On the training side, the official unit is $100.00 per hour, not per token. One hour of training costs $100, which buys roughly 25 MTok of input on that same fine-tuned model. At low volume, spending that money directly on input tokens is usually the better deal — which is exactly why step 4 of the tree gates on request volume.

The awkward reality: OpenAI is winding down self-serve fine-tuning

Anyone making a 2026 platform decision needs this. OpenAI's Deprecations page gives the schedule:

Models you already trained keep serving inference until their base model is retired. So this is not "your deployed service gets switched off"; it is "your ability to change it gets frozen."

The second row is the nasty one, because the gate is tied to your own usage. A project that fine-tuned heavily in 2025 and then went quiet for three quarters can come back to retrain and find the door shut by its own 60 days of silence, not by the 2027 date. If fine-tuning is in your stack, per-model usage monitoring is capability insurance.

Troubleshooting: four errors traced to source

Error 1: target modules not found

The message is Target modules ['q_proj', 'k_proj'] not found in the base model. Please check the target modules and try again. It comes from peft's tuners/tuners_utils.py and raises NoMatchingPeftModuleError. The cause is that many models fuse attention: Gemma, Phi, and some Qwen variants use a single fused qkv_proj matrix, so there is no separate q_proj, k_proj, or v_proj to target. Copying a Llama recipe hits this immediately. Two fixes: set target_modules="all-linear" and let peft match every linear layer, or walk model.named_modules() and print the real names first.

Error 2: cannot merge on a quantized base

The message is ValueError: Cannot merge LORA layers when the model is gptq quantized, from peft/tuners/lora/model.py. Training LoRA on a GPTQ-quantized base works fine, but merge_and_unload() fails, because quantized weights cannot simply be added to the adapter delta. In quantization plus LoRA setups, keep the adapter structure at inference instead of merging.

Error 3: cannot merge when layers are replicated

The message is ValueError: Cannot merge LORA layers when base model layers are replicated, from an adjacent branch of the same file. Once tensor parallelism has replicated layers, merging fails. If you hit this in a multi-GPU deployment, merge first and shard after, not the other way around.

Error 4: gradients die at the first layer

The message is RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn. It shows up when you enable gradient checkpointing to save memory: the frozen base produces outputs with no gradient, so backprop dies at layer one. The standard fix in PEFT setups is model.enable_input_require_grads(), which reattaches gradients to the input embedding output. This one misleads badly — it points at tensor 0, but the root cause is checkpointing combined with frozen parameters.

One more issue that is not an error but costs people a day: you train LoRA, load the adapter, and the output is identical to the base model. Usually you forgot model.eval(), or in a multi-adapter setup you never called set_adapter(). Run print_trainable_parameters() to confirm the adapter is actually attached before you start doubting the training run.

When to go back and fix the prompt instead

The cheapest regression path I found: when a fine-tuned model falls behind on new data, do not retrain yet. Write the constraints it learned back into the system prompt as a few sentences and run the comparison on the same base model. If the gap is inside your tolerance, retire the fine-tune — especially with the platform window measured in months, removing a dependency that is about to expire is itself a win.

FAQ

Can fine-tuning teach a model new knowledge?

Not recommended. The RAG paper is explicit that parametric models are limited in their ability to access and precisely manipulate knowledge, and that provenance and knowledge updates are open problems for them. If the knowledge changes, use RAG.

What rank should LoRA use?

Hugging Face's official PEFT documentation says start at 8 or 16, with lora_alpha typically set to twice the rank. Covering more target_modules increases adaptation capacity but also memory. The script above takes an r value you can change and rerun.

Do prompt caching and RAG conflict?

No, they stack. Put the retrieved chunks in the prompt prefix, and the part of the prefix that does not change can hit cache.

Is fine-tuning still worth investing in?

Depends on volume and task width. Narrow task, high volume, rigid output format: still yes. But pick a platform that will not close in six months, which usually means an open-weights base you host yourself.

Sources (all verified as of 2026-10-14)

• Anthropic official pricing docs (cache multipliers, Batch discount, data residency multiplier): https://docs.anthropic.com/en/docs/about-claude/pricing
• Anthropic official prompt caching docs: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
• Anthropic API release notes (2026-10-07, Sonnet 5.5 cache read price cut): https://docs.anthropic.com/en/release-notes/api
• OpenAI official pricing (Finetuning section, o4-mini-2025-04-16 tiers): https://developers.openai.com/api/docs/pricing
• OpenAI official deprecations (self-serve fine-tuning schedule): https://developers.openai.com/api/docs/deprecations
• LoRA paper: https://arxiv.org/abs/2106.09685
• RAG paper: https://arxiv.org/abs/2005.11401
• Hugging Face PEFT documentation: https://huggingface.co/docs/peft

Further reading on this site:

• Why RAG Retrieval Is Wrong: Three Layers and Five Real Errors
• vLLM 0.31.0 KV Cache: Paged Blocks, Measured, and Four Real Errors
• Attention From Scratch: NumPy Code Checked Against PyTorch 2.14.1

👉 Join Xiaomi MiMo Platform: Leading AI model platform with cost-effective inference

👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation

📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews