The Provenance Gap: Why Real-World Data Passes Technical Review and Fails Regulatory Review

A dataset arrives. The technical review goes well. Field completeness is high. Record counts meet the requirement. The mapping runs cleanly. Everyone signs off.

Eighteen months later, the same dataset creates a problem in a submission. Not because the values were wrong. Because nobody can explain where they came from.

Provenance is the reliability test that gets assumed

Our fitness-for-use framework covers the four dimensions of reliability that determine whether you can trust what a dataset reports: provenance and traceability, accuracy, completeness, and conformance. Three of those get tested during evaluation. Accuracy has a validated rate. Completeness has a percentage. Conformance has a common data model to check against.

Provenance is different. It rarely produces a number, so it rarely gets tested. It gets assumed, and the assumption holds right up until a reviewer asks how a value was derived.

This post is about that fourth dimension specifically. What documented provenance requires, why it degrades as data moves through intermediaries, and how to test for it before a dataset is under contract.

Four layers of a defensible data lineage

Provenance is the traceable history of a data element from creation to analysis. Four layers make it defensible.

Origin. Which system created this record, and in what clinical context? A lab value from an integrated in-house analyzer and a lab value transcribed from a faxed outside report are different data, even when they contain the same number.

Custody. What systems handled the data between origin and delivery? Every transfer is a point where meaning can shift. Undocumented transfers are unexaminable transfers.

Transformation. What was changed, and by what rule? Normalization, code mapping, unit conversion, and de-duplication all alter data. Each transformation needs a documented rule and a preserved original.

Consistency. Did capture practices hold steady across contributing sites and across time? A variable captured reliably at 40 sites and sporadically at 160 others is a bias problem wearing the costume of a completeness statistic.

That last layer is the one most often missed, and it is where provenance and completeness intersect. Aggregate completeness percentages hide site-level variation. A field that reads 85 percent complete across a network could be 100 percent complete at some sites and near zero at others. Those two scenarios support entirely different analyses, and the aggregate number does not distinguish them.

Why observational research raises the stakes

A clinical trial collects data on purpose. The protocol defines what gets recorded, when, and by whom. A non-interventional study, meaning one that observes care as it happened rather than assigning treatment, inherits data that was collected to treat patients and bill insurers. Nobody recorded it with your research question in mind.

That inheritance is what makes provenance load-bearing. You did not control how the data came to exist, so you have to be able to describe how it did. Postmarketing safety work runs almost entirely on this kind of evidence, which is why documentation requirements have tightened around it. We covered that shift in What ICH M14 Changes About How Sponsors Choose Real-World Data Sources.

Why aggregation strips lineage

Real-world data reaches sponsors through several paths, and the paths differ in how much history survives the journey.

Data assembled through layers of intermediaries tends to arrive stripped. Each handoff normalizes, and normalization discards context. By the time a dataset has passed through a clearinghouse, an aggregator, and a vendor mapping layer, the record of what happened at the point of care is often gone. What remains is a clean table with no biography.

That works until someone asks a question the table cannot answer. Regulatory review asks those questions routinely. FDA described its approach in the December 2025 final guidance on real-world evidence for medical devices: relevance and reliability assessed case by case, based on the scientific strength and quality of the evidence. On the drug and biologic side, ICH M14 requires sponsors to document data source selection, variable definitions, and the handling of bias. Both point at the same underlying requirement. You have to be able to show your work.

Direct integration preserves more. When data move from an EHR through a documented pipeline to a defined output, each step can be recorded and audited. The transformations are known because they were designed rather than inherited.

A data dictionary looks the same either way. The difference surfaces when a reviewer asks how a variable was derived and the answer takes six weeks to assemble.

The site relationship is a data quality control

Provenance depends on knowing your sources. Not their count. Their identity.

When contributing sites are named and participating by choice, several things follow. Capture practices can be documented, because someone can ask. Anomalies can be investigated at the source. Changes in workflow that affect data can be identified when they happen rather than discovered during analysis. The chain of custody starts with a known party.

When sites are anonymous, none of that holds. Questions about the data have no addressee.

This is why BEKhealth structures site participation in real-world data as opt-in. Every contributing site is a known participant. The network spans more than 30 million patients across 200 research sites, with over 20,000 HCPs and more than 1,000 trials of collective research experience, and every dataset traces back to identified sources with documented integration.

Research-ready data is data with a documented history. The history has to be built during collection, because it cannot be reconstructed afterward.

Three questions that test for provenance

Standard evaluation covers accuracy rates, completeness, and conformance. These three test the dimension that usually goes unexamined.

Can you produce a data lineage document showing the path from source system to delivered dataset, including every transformation rule applied?
Is completeness reported at the site level, or only in aggregate across the network?
How are changes in site capture practice detected and documented over time?

A data partner who can answer all three is offering a dataset you can defend. A partner who cannot is offering a dataset you can analyze, which is a different and smaller thing.

The cost of finding out late

Provenance problems have a characteristic timing. They stay invisible during acquisition and analysis, then surface during submission preparation, when the study is designed, the analysis is run, and the timeline has no slack.

At that point the options are poor. Reconstruct the documentation, which is slow and often incomplete. Narrow the claim to what the data can support. Or absorb the review risk.

The work that prevents all three happens at selection. The iterative approach EMA describes in the M14 guideline puts data fitness assessment and feasibility ahead of design for exactly this reason. Ask the lineage questions before the data arrive, and provenance becomes a property of the dataset rather than a project.

Further reading

For a detailed analysis of FDA’s final guidance on real-world data and real-world evidence for drug and biological products, see this summary from RTI Health Solutions. BEKhealth provides the underlying real-world data for collaborative research programs with RTI Health Solutions, which contributes health economics and outcomes research expertise.