In this article
Synthetic data is generated data used for a development or evaluation purpose. It can be useful for drafting examples, exploring variations or investigating a rare scenario. The question is how the team knows the generated material is suitable for the job.
The risk is not that every generated example is bad. It is that an apparently efficient pipeline can repeatedly reproduce its own assumptions while losing contact with the work customers actually do. A very large dataset can still contain a very small range of ideas.
What model-collapse research does, and does not, show
A 2024 Nature paper by Shumailov and colleagues examines how indiscriminate recursive use of model-generated training data can degrade subsequent models and lose parts of the original distribution. This is a specific research finding about feedback through training, not proof that every use of synthetic data makes a model worse.
Another 2024 study, Gerstgrasser and colleagues’ work on accumulating real and synthetic data, reports that preserving original real data while accumulating synthetic generations avoided collapse in its studied settings. Those conditions matter. Neither paper supplies a universal safe synthetic-data percentage for your product.
My business inference is straightforward: preserve an independent connection to the task and test the proposed mixture. Treat generation, selection and evaluation as separate responsibilities. A synthetic pipeline needs evidence about its output, not just evidence that it ran.
Useful generated examples have a defined purpose
Imagine a fictional team building an assistant for maintenance requests. It already has authorised examples of real requests. The team might generate alternative phrasings to test whether the system recognises the same issue when a user writes briefly, misspells a term or describes symptoms indirectly.
That is a bounded purpose. It differs from asking a model to invent thousands of “typical customers” and then treating their invented needs as market research. Generated customers do not buy subscriptions, complain about missing features or choose a competitor. They cannot replace discovery with actual people.
Write a short purpose statement before generating anything: what variation is missing, how it will be checked and what claim the new examples will support. Also state what they cannot establish. This keeps a convenience tool from quietly becoming the entire evidence base.
Watch for diversity that is only cosmetic
A model can rewrite the same scenario in many voices without adding meaningful coverage. Changing names, sentence length and tone may leave the underlying assumptions untouched. A dataset can look impressively varied while never including an incomplete record, an unusual workflow or a customer who does not fit the imagined average.
List the dimensions that actually change the task. For maintenance, those might include missing access information, conflicting reports, urgent damage or an unclear responsibility boundary. Have someone familiar with the work review whether the examples represent those differences. Make the difficult cases realistic rather than merely dramatic.
Keep the exam out of the training room
Hold out independently selected examples that the team does not use for generation templates or iterative tuning. Where real data is appropriate, use only material approved for that purpose and handle sensitive information under the organisation’s rules. Synthetic material is not automatically anonymous just because a model produced it.
Compare a baseline with and without the new synthetic data. Examine the intended improvement and regressions on other task groups. If a generated batch improves format compliance but worsens handling of uncertain requests, that is a product decision to make visible.
My business evaluation guide explains why a single average can obscure this tradeoff. The result you want is a more useful system, not a larger training directory.
Document the recipe and its limits
Keep the generator version, prompt template, source material, sampling approach, filtering rules and review decisions. Mark generated records so they can be identified later. Do not erase provenance when combining them with human examples.
Hugging Face’s dataset-card documentation is a useful starting point for describing a dataset’s contents and context. For a synthetic collection, I would also record the intended use and the checks that support it. That extra receipt is my proposed practice, not a claim that a platform label validates quality or rights.
Choose the cheap experiment that can prove you wrong
Start with one known gap. Generate a manageable batch, review it and measure whether it helps on independent cases. Keep the rejected examples and reasons long enough to understand the failure patterns within your approved retention process. A model that consistently invents an impossible business rule is teaching you something about the generation setup.
Before scaling, ask whether a clearer instruction, better retrieval or improved source record solves the problem more directly. See fine-tuning versus RAG for that diagnosis and official data sources for provenance questions.
I help teams make this kind of evidence useful to product and management decisions. The goal is to use generation where it earns its place while keeping real users, independent checks and accountable owners in the process. The photocopier can assist the department. It should not become the department’s only witness.
Questions people ask
Does synthetic data always cause model collapse?
No. Research findings depend on how data is generated, selected, retained and reused. Indiscriminate recursive replacement differs from carefully evaluated use alongside original real data.
Can synthetic customers replace customer research?
Generated personas can help explore assumptions, but they do not establish real customer demand. Validate important product and buying assumptions with actual customers and observed behaviour.
How should a team test synthetic training data?
Define the intended improvement, document the generation process, keep independent evaluation cases and compare a baseline with and without the synthetic data. Inspect regressions by task type.
Something here sounds like your workplace?
Define the use case, connect approved systems and test a bounded workflow before expanding it.