Fine-tuning or RAG? The expensive mistake happens before training

Fine-tuning changes how a model behaves through additional training. Retrieval-augmented generation, usually called RAG, supplies relevant information when a request is made. A business can need either, both, or neither. The useful first question is which failure is preventing the user from completing the job.

Imagine a support assistant that politely quotes an old cancellation policy. More examples of polite responses will not fix the missing current policy. Now imagine an assistant that has the right policy but repeatedly puts answers into the wrong classification format. That is a different investigation. Buying training before naming the failure is like sending the office printer on a communication course.

Separate missing knowledge from inconsistent behaviour

OpenAI’s accuracy guide distinguishes context problems from behaviour problems. I use that distinction as a starting diagnosis, then test it against actual work. It is a useful lens, rather than a guarantee that every difficult case belongs in one box.

A permission failure is especially easy to misdiagnose. Training on everyone’s files does not create a reliable per-user access-control system. The process must decide what this person may see on this request. My separate guide to permissions in an internal AI knowledge assistant covers that part of the design.

Make the baseline boring enough to trust

Write the task in ordinary language. Define a good answer, an unacceptable answer and a situation in which the system should ask for help. Try a clear prompt with a small number of relevant examples before starting a training project. Save the results and the exact setup, including model version and any tools used.

Then test a representative holdout: cases that were not used to write the prompt or build training examples. Include short requests, messy requests, unfamiliar customers and the inconvenient exceptions people encounter at work. A tidy demonstration set tells you whether the demonstration works. It cannot by itself tell you whether the product works.

For a hypothetical invoice-routing task, the target might be a correctly assigned queue with a reason a reviewer can check. A plausible paragraph is not the outcome. If the system cannot identify the supplier because the source record is incomplete, the next improvement may belong in the record system, not in the model.

When fine-tuning deserves a serious experiment

Consider tuning when a stable, repeated behaviour matters and you have suitable examples you are entitled to use. Examples might show the classification conventions, domain language or response structure the business expects. The examples need consistent labels. A thousand contradictory instructions are an office argument in a different file format.

Training is also an operational commitment. Someone must curate examples, protect evaluation data, run comparisons and maintain the result when the base model or workflow changes. The comparison should include the simplest acceptable alternative. A model that saves a little inference cost but creates a large maintenance burden may be the wrong business decision.

When retrieval is the stronger first move

Investigate retrieval when information changes, needs traceable sources or differs by customer and access rights. Test source quality, chunk selection and the final answer separately. If retrieval returns the wrong document, polishing the answer generator can produce a more persuasive mistake.

Keep a visible boundary between retrieved content and instructions. A document can contain useful facts without having permission to tell your system what actions to take. Also test the case where the answer is absent. A clear, bounded refusal can be a better outcome than an invented answer delivered with perfect grammar.

Check whether the training route still exists

Platform availability matters. As checked on 26 October 2026, OpenAI’s supervised fine-tuning documentation says its hosted fine-tuning platform is winding down and is unavailable to new users. That is a provider-specific lifecycle change, not the end of fine-tuning as a technique. Verify eligibility and shutdown dates before designing around any hosted training service.

The broader lesson is to keep your examples and acceptance tests portable. My article on fine-tuned model migration explains how to turn that dependency into a plan rather than a surprise.

Approve the next experiment, not an architectural identity

Give the team a bounded decision: compare the baseline, a retrieval change and, where justified and supported, a tuning experiment. Record quality by case type, cost per accepted outcome, latency and reviewer effort. Agree beforehand what improvement would justify the extra complexity.

I help teams connect those choices to the product and the people operating it. A useful first conversation starts with examples of what currently fails, the decision the system supports and who owns the result. “We need our own model” can wait until the evidence gives it a job.

Questions people ask

What is the business difference between fine-tuning and RAG?

Fine-tuning changes model behaviour through additional training. RAG supplies relevant information during a request. Diagnose the failure and compare both with a clear baseline.

Can fine-tuning replace document permissions?

Do not rely on model training to enforce per-user document access. Enforce permissions in the application and retrieval path, and test for disclosure failures.

Should a new OpenAI project start with hosted fine-tuning?

Check current availability first. On 26 October 2026, OpenAI’s documentation says its hosted fine-tuning platform is winding down and is unavailable to new users.