Better Data Beat Better Architecture: What That Means for Your Company

Boards keep asking for “our own GPT.” The invoice they have in mind is a from-scratch pre-train on company files.

On 8 September 2026, Dwarkesh Patel and Jerry Han published a 2019–2025 pre-training experiment. They mixed year-by-year model recipes with year-by-year public data sets, up to 1e19 FLOPs. Better data won on compute efficiency. For a mid-size firm that still means something plain: clean the corpus you already have, then retrieve from it. Do not fund a shrine to weights.

Quick answer

Usually do not train your own frontier model. For a mid-size Australian or Singapore company in 2026, the useful stack is a frontier API or Bedrock in your account, plus retrieval over clean, permissioned data. Patel and Han found better data gave 12.0× compute efficiency at 1e19 FLOPs; better recipes gave 3.7×. Proprietary data is a moat until a lab absorbs the same facts. BloombergGPT’s finance pre-train looked valuable, then GPT-4 beat it on public finance tasks within months.

Best for: founders and ops leads who must choose train-own-weights versus a frontier API plus RAG, and who need a shelf-life warning, not another “data is the new oil” essay. Honest limit: Cipher Projects is an Australian engineering studio. We ship private AI and production agents. We do not pre-train a 50-billion-parameter model, and we are not an assessor.

Last updated: 12 September 2026. The scarce thing on our jobs is a specified problem, clean proprietary data inside a boundary you control, and an evidence trail. See the cluster glossary if a term on this page is being used as atmosphere.

Should a mid-size Australian company train its own model on internal data?

No. A from-scratch pre-train is the wrong default for a mid-size Australian or Singapore firm in 2026.

Train your own weights when three facts are all true: the corpus cannot be retrieved as documents, a lab cannot buy or scrape the same facts, and you can re-train when the next public frontier model lands. That set is rare. Most “company data” is contracts, tickets, wikis, mail, and warehouse rows. Those are retrieval problems with a permission model. They are not a reason to spend on a ten-million-dollar pre-train.

A specialist model can still jump on one use case if the data is unique and clean. The jump dies when the next general model sees the same mutual information, or when your mix is junk. Keep the learning loop: ops data in, evals out. Do not keep a museum of weights.

What did the 2019–2025 experiment actually measure?

At a 1e19 FLOPs budget, better data gave a 12.0× compute-efficiency gain and better model recipes gave 3.7×. Data won by 3.24×.

Patel and Han trained mixes of a year’s public recipe and a year’s public corpus. The 2019 recipe was GPT-2. The 2025 recipe was OLMo-2. The 2019 corpus was OpenWebText: about 9 billion tokens of web pages linked from Reddit with enough upvotes. The 2025 corpus was UltraFineWeb: a much larger scrape plus a classifier that predicts which text will help. They scored end skill on OLMES, a bundle of ten easier benchmarks, mostly multiple-choice QA. They could not use a single held-out loss, because the data sets themselves changed.

Claim (1e19 FLOPs)	Published figure
Data compute-efficiency gain, 2019→2025	12.0×
Recipe / architecture gain, 2019→2025	3.7×
Data advantage over recipes	3.24×
OLMES variance from additive data + model effects	88%

A linear fit of OLMES score = mean + model effect + data effect explained 88% of the variance at 3.16e18 FLOPs. Recipe gains and data gains mostly stacked. You did not need a special pile to unlock a given recipe.

Read the limit on the same page. This is a small-scale run. The authors say model work still mattered because it made larger compute usable: longer context, stable training, mixture-of-experts, FlashAttention, and the rest. They also say data quality may matter less once a model is huge, because a large net can swallow a dirtier mix. Year-on-year they measured 1.51× on the data side and 1.24× on the model side, jointly 1.57×, which is well below some software-progress estimates. If you only remember 12.0×, remember the budget it was measured at.

Transformer papers still get copied in months. Data pipelines stay inside the lab. Your next 10×, if it exists, is more likely on a shared drive than in a new attention paper. If the next question is what that drive costs to serve, read why memory, not GPU count, is the inference bottleneck.

Why does junk in the training mix still show up in the model?

Pre-training cooks whatever you feed it. A bad mix teaches bad habits. Post-training only garnishes the result.

The Pile (Gao et al., 2020) is an 825 GiB English corpus from 22 sources: web crawl, PubMed, arXiv, GitHub, law, patents, and more. The point was diversity, not a tight filter. Patel and Han found The Pile did much worse than OpenWebText on OLMES. That is not a shock if the eval is English web-prose multiple choice and a large share of tokens never transfer. What’s In My Big Data? (Elazar et al., 2024) later counted more than 60% duplicate documents in The Pile, mostly from oversampling. RedPajama and LAION-2B-en sat near 50% duplicates. Repeated punctuation was often the most common n-gram.

LAION-5B is the image-mix warning. It is a Common Crawl link set, not a store of pixels. In December 2023 the Stanford Internet Observatory reported 1,008 links to known or likely child sexual abuse material. LAION took the set down. In August 2024 it published Re-LAION-5B after matching known-bad hashes. The company lesson is smaller than the scandal: an unvetted scrape is not a corpus. You inherit the junk and the legal risk.

If your internal mix is tickets, Slack, and half-empty Confluence, a pre-train will memorise that voice. Retrieval over a cleaned, permissioned slice will not. Distillation compresses digested data. It does not invent a source of truth you never wrote down.

Did the Hutter Prize also reward data order, not only better compressors?

Yes. Recent wins spent serious work on the order of Wikipedia articles, not only on a smarter compressor.

The Hutter Prize pays for lossless compression of enwik9, the first 1 GB of English Wikipedia. The current public bar is to beat 110,793,128 bytes. Kaido Orav and Byron Knoll’s fx2-cmix took the September 2024 record. The fx2-cmix README lists an updated article order as a major change: embed each article, reduce to one dimension, cluster, reverse, then move image and disambiguation pages by hand. Starlit, the 2021 winner, had already reordered articles so similar pages sat together. Context models compress better when the next page looks like the last one. That is a curriculum. It is the same lesson as a clean pre-train mix, at 1 GB.

What is the shelf life of proprietary data as a moat?

Months to one product cycle on public tasks, not a decade. BloombergGPT is the warning.

In March 2023, Wu et al. published BloombergGPT: a 50-billion-parameter model trained on FinPile plus public text. FinPile was about 363 billion finance tokens from Bloomberg’s archives. Public data added about 345 billion tokens. The paper said the mix beat similar-size open models on finance tasks and stayed competitive on general NLP. That looked like a moat. The firm had forty years of dated filings, news, and rights-cleared text.

In May 2023, Li et al. compared ChatGPT and GPT-4 to BloombergGPT on public finance NLP. GPT-4 won on almost every data set they ran except named-entity recognition. On FiQA sentiment, 5-shot GPT-4 scored 88.11 weighted F1; 5-shot BloombergGPT scored 75.07. On ConvFinQA, zero-shot BloombergGPT was 43.41%; ChatGPT was already 59.86%, and GPT-4 was higher still. The authors took BloombergGPT’s numbers from the original paper. They still concluded that a general frontier model was the first choice for those public finance tasks.

Two facts can both be true. Internal Bloomberg tasks may still prefer the in-house model. On the public benchmarks other people can run, the specialised pre-train lost its lead in weeks. Proprietary data is a moat until it is mutual information a lab already absorbed, or until a larger general model makes the specialised weights look small. If your “unique” corpus is also in EDGAR, PubMed, or last year’s web scrape, assume the shelf is short. If the corpus is live operations data behind an ACL, retrieve it. Do not bake it into weights you will not re-train.

Where those weights may legally sit is a different page: frontier model weights now have borders.

Should you train your own model, fine-tune, or use RAG?

Default to a frontier API plus RAG on clean, permissioned data. Fine-tune when the style or format is stable. Pre-train only when the facts cannot be retrieved and will not be in the next public model.

Option	What you buy	Best for	Honest limit
Train your own model	From-scratch or continued pre-train on internal tokens	A corpus a lab cannot buy or scrape, plus a budget to re-train when the frontier moves	Cost, talent, and shelf life. BloombergGPT lost the public-benchmark lead in months. Most AU/SG firms do not have FinPile.
Fine-tune	An adapter or a full fine-tune on a frontier or open model	A stable style, a fixed form, or tool traces you can score	Facts that change still need retrieval. A fine-tune will drift. You still own evals.
RAG + frontier API	Retrieval over permissioned docs plus a hosted model	The usual mid-size job in 2026: contracts, tickets, wikis, mail, warehouse rows	Retrieval quality and permissions are the product. A bad index looks like a dumb model.
Cipher private AI on Bedrock	The RAG path in your AWS account, with evals and a boundary	Australian and Singapore teams that want the stack operated, not a shrine to weights	We do not train a BloombergGPT-class model. We are not an assessor. We prepare systems and evidence.

Managed Bedrock Knowledge Bases, a customer-managed index, and full DIY RAG are three ownership paths, not one brand. The cutover lives in Bedrock Knowledge Bases vs custom RAG. Do not DIY retrieval to feel senior. DIY when you have a measured failure Managed KB cannot fix: ACLs, hybrid search quality, multi-tenant isolation, or a non-Bedrock runtime.

If the product is spatial or robotic, a chatbot-plus-RAG stack may be the wrong bet. That split is on world models versus chatbots. If the scarce skill is naming the task for a swarm of cheap agents, start with what companies do after a Millennium Prize claim.

What does Cipher build instead of a ten-million-dollar pre-train?

We put a frontier model behind Bedrock, or an equivalent API, in the customer’s account, then retrieve from a cleaned, permissioned corpus with citations and an eval set the business owns.

That is the private-AI pattern we actually ship. Data stays in the account. Sydney or Singapore Region is a first-class choice, not a slide. Connectors land in a Knowledge Base when they fit. When they do not, we keep the vector store and the chunker. Every answer that matters has a source span and a score. Ops data comes in. Failures go back into the eval set. The model is rented. The loop is yours.

We keep seeing the “train our own GPT” request. The useful reply is a permission matrix and a scored task, not a GPU quote. A pre-train burns the budget on weights you will not re-train when GPT-class models move again. RAG on dirty SharePoint burns the budget on an index that cites the wrong file. The work is the clean slice, the ACL, and the trail an auditor can read. Wider rollout notes sit in the enterprise AI implementation guide. Lab-risk talk sits on what p(doom) means for a company, not on this page.

What should you do this quarter?

Name one job, clean the files that job needs, put them behind an ACL, and score the model against a set you own.

Write the job as a question a person already asks, with a right answer you can check.
List the systems of record for that job. Drop chat logs that are not a source of truth.
Build a permission matrix before you chunk. If a person cannot open the file, the model cannot either.
Stand up retrieval in your account. Bedrock Knowledge Bases if the connectors fit; custom RAG when they do not.
Keep an eval set the business edits. Ops data in, misses out. That is the loop. The weights are rented.

Start with a permission matrix and an eval set on the job you actually run. Do not start with a GPU quote for a model you will not re-train.

FAQ

Should a mid-size Australian company train its own model on internal data? No, not as a from-scratch pre-train. Use a frontier API or Bedrock plus RAG on clean, permissioned data. Train weights only when the corpus cannot be retrieved, a lab cannot buy the same facts, and you can re-train when the frontier moves.

Should we use a frontier API plus RAG instead? Yes, that is the usual 2026 stack for Australian and Singapore mid-size work. The product is retrieval quality and permissions, not a new architecture.

What is the shelf life of proprietary data as a moat? On public tasks, weeks to months. BloombergGPT’s paper is dated 30 March 2023. By May 2023, Li et al. had GPT-4 ahead on most public finance NLP they ran. Live operations data behind an ACL lasts longer, because a lab cannot scrape it. Bake that into weights only if you will re-train.

When is fine-tuning the right move? When the style or form is stable and you can score it: a fixed memo layout, a tool trace, a house voice. Fine-tuning does not replace retrieval for facts that change.

Did better data really beat better architecture? On the published 2019–2025 run, yes, at 1e19 FLOPs: 12.0× from data, 3.7× from recipes, 3.24× advantage, 88% additive. The authors warn the run is small and that model work still unlocked scale. Use the numbers with that budget attached.

What will Cipher not do? We will not sell you a BloombergGPT-class pre-train. We will not certify your organisation. We build the boundary, the retrieval path, and the evidence trail. If that is the job, applied AI engineering is the offer.

Sources

Patel and Han, 8 September 2026: “Pretraining progress is mostly coming from data.” 12.0× data vs 3.7× recipes at 1e19 FLOPs; 3.24× advantage; 88% additive OLMES variance; OpenWebText → UltraFineWeb; GPT-2 → OLMo-2; The Pile underperforms OpenWebText on OLMES.
Wu et al., arXiv:2303.17564 (30 March 2023): BloombergGPT: 50B parameters; ~363B finance tokens plus ~345B public tokens.
Li et al., arXiv:2305.05862 (EMNLP 2023 Industry): ChatGPT and GPT-4 vs BloombergGPT on public finance NLP. GPT-4 leads on most tasks except NER; FiQA 5-shot weighted F1 88.11 vs 75.07; ConvFinQA BloombergGPT 43.41%.
Gao et al., arXiv:2101.00027: The Pile: 825 GiB, 22 sources, diversity-first mix.
Elazar et al., arXiv:2310.20707: What’s In My Big Data? More than 60% duplicate documents in The Pile; ~50% in RedPajama and LAION-2B-en.
Thiel and Hancock, Stanford Internet Observatory (20 December 2023): 1,008 links to known or likely CSAM in LAION-5B.
LAION, 19 December 2023: temporary takedown of LAION-5B. Follow-up: Re-LAION-5B (August 2024).
Hutter Prize: compress enwik9 (1 GB of English Wikipedia). fx2-cmix record 3 September 2024; article-order pipeline in the fx2-cmix README.

We did not find a public named source for an unnamed data subsidiary valued at tens of billions, so we dropped that anecdote. We did not find a citable Stable LM data-mix post-mortem that added facts beyond The Pile and LAION, so we did not name one.