As the parameter scale of large language models surpasses trillions, the focus of debate regarding data recipes has shifted from "quantity" to "quality and source." The weight game between synthetic data and real-world data is essentially a reassessment of the applicable boundaries of the scaling law.

Real-world data: Irreplaceable contextual anchors

Real-world data generally refers to the text, code, audio, and video transcription content generated by humans during natural interactions. Its core value lies in preserving pragmatic intent, social common sense, and cultural inertia. From a technical implementation perspective, large-scale collection of such data mainly relies on two types of channels:

• Public web crawling systems - Using Web Scraper API to incrementally scrape news portals, academic preprint platforms, and technical Q&A communities, followed by deduplication, security filtering, and copyright negotiation before injecting into the training corpus. Such services can bypass JavaScript rendering obstacles, manage proxy pools, and automatically parse structured content, supporting continuous updates of billions of documents while complying with robots.txt.
• User behavior logs - With authorization, de-sensitizing search engine clickstreams and smart assistant interaction records to mine implicit feedback signals.
• Integration of open-source datasets - Integrating authoritative sources such as Wikipedia, legal cases, and government open data.

The non-stationary characteristics of real data are the root of its irreplaceability. Language evolution, sudden events, and the emergence of new terms are all instantly reflected in real corpora.

Synthetic data: Controllability and scale bottlenecks

Synthetic data refers to samples constructed using methods such as rule templates, early model bootstrap generation, or reverse translation. The advantages are concentrated in the following dimensions:

• Decreasing marginal costs - Once a generation pipeline is established, it can produce millions of instruction pairs, especially suitable for cold starts in vertical domain multi-turn dialogues.
• Privacy avoidance - It does not contain real user identity information, avoiding compliance red lines like GDPR.
• Long-tail coverage - By intentionally combining low-frequency entities and logical chains, it compensates for the representation deficiencies of sparse events in real data.

However, synthetic data has inherent degradation traps. When models repeatedly consume their own generated outputs, the features of the true distribution are smoothed over iterations, ultimately leading to a collapse of tail diversity.

A comparative experiment conducted by Stanford University in 2023 indicated that models fine-tuned entirely on synthetic data had a 19% decrease in accuracy when handling counterfactual reasoning tasks compared to a pure real data baseline. Even more concerning is that synthetic data may amplify the positional biases of the seed model—if the teacher model has stereotypical associations with certain groups, the generated samples will reproduce that pattern more frequently.

Phase specificity: Differen0t roles of pre-training, fine-tuning, and alignment

There are essential differences in sensitivity to data types at different training stages.

1. Pre-training Stage

The absolute dominance of real-world data is hard to shake. At this point, it is necessary to learn lexical, syntactic, world knowledge, and reasoning chains, while the diversity of large-scale corpus cannot be replicated by synthetic generators. Currently, top foundational models use tens of trillions of tokens from real or nearly real documents (such as OCR-scanned books).

2. Supervised Fine-Tuning

The value of synthetic data intervention is highlighted during this stage.  High-quality human annotation is costly and difficult to scale, while synthetic instruction-answer pairs can be generated from a small number of seed samples using a teacher model, followed by length filtering and answer quality ranking, allowing task performance to quickly approach human annotation quality. Architectures like WizardLM and Evol-Instruct have verified the effectiveness of this approach.

3. Human Feedback Reinforcement Learning

Caution is needed when introducing synthetic preferences. Reward models must be built on real human preference comparison data, and any synthetic substitutes will introduce optimization target shifts. Currently, the industry practice is to mix 15%–25% of fresh real feedback samples in each iteration to anchor value orientation.

Data collaboration strategies: Technical routes of isolation and fusion

Simply debating which type of data is more important is inefficient. The current efficient data governance framework presents two technical routes:

Label isolation training: In the pre-training stage, real data is strictly used, while in the fine-tuning stage, synthetic data is confined to specific task domains and domain weights are frozen to prevent the forgetting of general capabilities.

Quality-aware mixing: Confusion detection and factual consistency verification are performed on synthetic samples, retaining only instances with a confidence level above the threshold; at the same time, real-world data is used as a validation set to dynamically adjust the sampling weights of the two types of data.

Ethical and Legal Compliance Considerations

When using Web Scraper API to collect real data, a copyright filtering layer must be established. Recent case law from the European Court has clarified that systematically scraping substantial parts of a database may constitute infringement. The solution is to only scrape sites that clearly adopt open license agreements or to obtain authorization for transcoding through content distribution alliances. Although synthetic data avoids original copyright issues, if its generation process relies on copyrighted text for distillation, there remains a risk of derivative infringement.

Conclusion

In the value chain of LLM training, real-world data determines the upper limit of model capabilities and adaptability to the times, while synthetic data provides leverage for fine-tuning and cost optimization. The two are not mutually exclusive options but form a second optimization dimension beyond the current parameter scaling through rigorous engineering overlap. The future competitiveness of the data flywheel will be reflected in the coupling depth of real corpus acquisition bandwidth and synthetic data quality gating mechanisms.