RAG, fine-tuning or just a better prompt?

The question behind the question

“Should we fine-tune a model on our data?” is one of the most common questions we get, and it is almost always the wrong first question. Not because fine-tuning is bad, but because the thing people usually want from it — the model should know about our business — is not what fine-tuning is for.

There are three distinct techniques here and they fix three distinct problems. Choosing between them gets much easier once you can say which problem you actually have.

• Prompting changes what the model is asked to do.
• Retrieval (RAG) changes what information the model has when it answers.
• Fine-tuning changes how the model behaves by default.

Most business problems that look like AI problems are the second kind, a fair number are the first, and a genuinely small share are the third.

Prompting: start here, and stay longer than feels right

Prompting has a reputation problem. It sounds like the trivial option, so teams skip past it toward something that sounds more like engineering.

That instinct is expensive. A large share of “the model isn’t good enough” turns out to be an instruction problem: the task was described vaguely, the output format was left implied, the edge cases were never mentioned, and no examples were given. Modern models follow detailed instructions well, and the difference between a two-line prompt and a carefully specified one with three worked examples is frequently larger than the difference between two model generations.

Prompting is the right tool when the model has the knowledge and the capability, but is not reliably doing the thing you want — wrong tone, wrong format, ignoring a rule, being verbose, or handling an edge case badly.

Its limit is knowledge. No amount of instruction tells a model what is in a document it has never seen. When the failure is “it doesn’t know about our products, our policies, our customers,” prompting cannot fix it, and that is where retrieval starts.

Practical note: prompts are code. They belong in version control, they need the same review as anything else that changes behaviour in production, and they need an evaluation set — otherwise every improvement is also an unmeasured regression somewhere else.

Retrieval: the answer to “it doesn’t know our stuff”

Retrieval-augmented generation is a straightforward idea wearing an intimidating name. Before asking the model to answer, you search your own content for the passages relevant to the question, and you include those passages in the prompt. The model answers from material you supplied rather than from memory.

This is the right approach for the overwhelming majority of “AI over our own data” projects: internal knowledge bases, policy and documentation lookup, customer support grounded in real product information, research over a document corpus, anything where the answer should come from a source you control.

The reasons it wins are practical:

• The knowledge is current. Update the document and the next answer reflects it. No retraining, no delay.
• Answers can cite sources, which is what makes them checkable — and checkability is usually what makes a system deployable at all.
• Permissions are enforceable. You can restrict what a given user’s query retrieves, which you fundamentally cannot do once information is baked into model weights.
• It fails visibly. If retrieval finds nothing relevant, the system can say so instead of inventing something.

The catch is that RAG is a retrieval problem before it is an AI problem, and this is where implementations go wrong. If the search step returns the wrong passages, a better model will produce a more articulate wrong answer. The work that determines quality is unglamorous: how documents are split, how they are indexed, whether keyword and semantic search are combined, whether results are re-ranked, how tables and images are handled, and how the source content is kept clean in the first place.

That last one deserves emphasis. A retrieval system built over a document store containing three versions of the same policy will confidently produce answers from the wrong one. Retrieval quality is downstream of content quality, and this is often where a data engineering problem hides inside what was scoped as an AI project.

Fine-tuning: for behaviour, not for facts

Fine-tuning continues training a model on your examples so that its default behaviour shifts. It is a real tool with real uses — they are just narrower than the enthusiasm suggests.

Fine-tuning is a good fit when:

• You need a consistent style or format that is hard to specify but easy to demonstrate, and you have hundreds or thousands of examples of it.
• The task is narrow, repetitive and high-volume — a specific classification or extraction job — and you want a smaller, cheaper, faster model to match a larger one’s quality on that one task. This is often the strongest commercial case: not better output, but equivalent output at materially lower cost per call.
• The domain has genuinely unusual language that general models handle poorly, and prompting with examples has been tried and is not enough.

Fine-tuning is a poor fit for teaching a model facts. Facts learned this way cannot be cited, cannot be permission-scoped, cannot be corrected without retraining, and are not reliably recalled — the model may produce something adjacent to what you taught it, delivered with full confidence. If a fact changes weekly, encoding it in weights is the wrong place to put it.

The costs are also real: a curated training set, an evaluation set to prove it helped, infrastructure to serve the model, and a commitment to redo the work when you want to move to a newer base model. That last cost is easy to miss and is the one that bites: fine-tuning ties you to a snapshot of a field that moves quickly.

A practical order of operations

The sequence below is roughly increasing in cost and decreasing in frequency of being the right answer.

1. Write a proper prompt. Detailed instructions, output format specified, three to five worked examples, explicit handling for the awkward cases. Measure it against a real evaluation set.
2. Check whether it is a knowledge problem. If failures are the model not knowing something, no prompt fixes that — go to retrieval.
3. Build retrieval, and invest in the retrieval half. Get the right passages in front of the model, with citations. Most projects should stop here, and most that fail here fail on search quality or content quality rather than on the model.
4. Fix cost and latency structurally. Route easy cases to a smaller model, cache what repeats, retrieve less and better. This is where most cost problems are solved.
5. Consider fine-tuning for the narrow, high-volume, behaviour-shaped remainder — or when a small tuned model can replace an expensive general one on a well-defined task.

These are also not exclusive. The common production shape is a well-engineered prompt over a good retrieval system, with a small fine-tuned model handling one high-volume subtask. The techniques compose; the mistake is substituting one for another.

How to tell which problem you have

A quick diagnostic. Take twenty real failures and sort them:

• “It didn’t know that” — the information exists somewhere in your organisation and the model had no access to it. That is retrieval.
• “It knew, but did it wrong” — wrong format, wrong tone, ignored a rule, too long, missed an edge case. That is prompting, and if it persists across hundreds of examples with no describable rule, it may be fine-tuning.
• “It made something up” — usually retrieval with citations, plus permission to say “I don’t know.” Sometimes it means the question was outside the system’s scope and the scope needs stating.
• “It was right but too slow or too expensive” — an architecture problem: smaller models for easy cases, better retrieval so prompts are shorter, caching.

That sort takes an afternoon and is worth more than a month of speculation about model choice.

The part that outlives the decision

Whichever technique you pick, two things determine whether the result is any good, and neither is the technique.

The first is evaluation. A fixed set of real cases with agreed correct answers, scored automatically, so you can tell whether a change helped. Without it, every one of the choices above is guesswork dressed as engineering.

The second is that the architecture should let the answer change. Model providers release meaningfully better models on a cadence shorter than most engagements. A system where the model is a swappable component — behind an interface, with an evaluation suite that can be re-run against a new one in an afternoon — keeps that as an upgrade. A system built around one provider’s specifics turns it into a migration project.

We build AI systems this way as a default: retrieval first, evaluation from the start, fine-tuning when the problem is genuinely shaped like it, and a boundary around the model so that next year’s better one is a swap rather than a rewrite. If you are trying to work out which of these your problem needs, that is a good use of a free initial consultation — we reply within 24 hours.