Executive Summary
Most large enterprises already have lineage. It usually lives in three places: a diagram generated during an audit remediation two years ago, the job dependency graph inside the ETL tool, and the parts of the estate one senior architect happens to carry in her head.
That arrangement held for a decade, because lineage was documentation. It described the estate to people who mostly already knew it.
Four things changed at roughly the same time:
- Auditors started asking how a reported figure was derived.
- BI platform consolidation put years of semantic modeling at risk.
- Cloud migration left most large estates running legacy and modern side by side, permanently.
- Agents began reading metadata to decide what a column means before answering a question.
Each of those promotes lineage into a control. A control has to hold under load, return the same answer twice, and answer as of a date someone else picks. Documentation never had to do any of that.
This paper covers the four jobs lineage now has to do, six ways the lineage most enterprises already own quietly fails at them, and a diagnostic you can run against your own estate this week with no vendor in the room.
Why Lineage Got Promoted
Lineage spent most of the past decade as a compliance artifact. It got built for a specific audit, satisfied that audit, and went stale. The staleness was tolerable because the diagram's job ended when the auditor left.
Four pressures ended that tolerance, and they arrived close enough together that most data teams felt them as one thing.
Audit questions moved from approvals to derivations
Governance programs were built to answer who approved a number. The questions arriving now ask how the number was produced.
Audit and disclosure work under SOX 302, BCBS 239, CSRD, and ISSB lands on data definitions: what a field meant, where it came from, and what happened to it between the source system and the filing. "We're working on it" gets expensive at that point, and the remediation narrative usually costs more than the finding.
Platform migration became a standing condition
BI consolidation moves faster than semantic modeling does. A team that spent years building models in one BI tool gets asked to reproduce them in another on a schedule set by a licensing decision.
The inventory question comes first. Which reports depend on which models, which models depend on which columns, and which of those columns still matter to anyone.
Lineage is how that inventory gets built. Migrations are also where teams find out exactly how much of it was missing, usually about three weeks in.
The estate stopped being one generation
Large estates run legacy and cloud in parallel now, and most will keep doing it indefinitely. A COBOL copybook feeds a warehouse table that feeds a cloud dashboard. A stored procedure untouched for a year still carries a definition three reports depend on.
Lineage that covers only the modern half of an estate covers half an estate. The half it skips tends to hold the oldest and most load-bearing business logic.
Systems started reading metadata before answering
This is the newest pressure and it's moving fastest. AI agents need metadata to generate an answer. Structured AI consumption through interfaces like Snowflake Cortex Analyst and Databricks Genie reads your metadata to decide what a column means, then answers a business question with it.
A human analyst who hits an ambiguous field name asks someone. An agent picks an interpretation and proceeds with confidence, right or wrong.
When the answer turns out wrong, lineage is the record that explains why. Metadata completeness becomes a property of the AI system itself, measured on the same terms as its accuracy.
The market moved at the same time
Two signals landed in the same window. Gartner revived the Metadata Management Magic Quadrant in November 2025. Categories don't come back when demand is flat.
The second signal is standardization. Open Semantic Interchange moved to the Apache Software Foundation as Ossie, now incubating with more than 50 participating organizations behind a YAML and JSON specification for exchanging semantic models, with converters shipping for formats including Snowflake Semantic Views. A shared file format settles how a definition travels. It leaves open who governs what goes in the file, where the definition came from, and whether the same definition reached every system that needed it. Those are lineage questions.
Where This Shows Up First
The promotion rarely announces itself. It shows up as four ordinary meetings that go worse than they should.
The incident review. A dashboard number moved overnight and no one on the call can say why. The investigation runs across three teams and takes four days, and most of that time goes to establishing which upstream objects were even in the path. The fix, once found, takes an hour.
The migration cutover. A BI consolidation is scoped from a report inventory, because that's the list that exists. Halfway through, the team finds that a third of the reports depend on models that were never catalogued, and some of those models depend on transformations that live in a stored procedure. The schedule was built on an inventory that was already incomplete.
The audit request. A regulator or an internal auditor names a figure and a date. The answer requires knowing what a definition meant on that date and which systems the data crossed to get there. What actually happens is a project: people pulled off roadmap work to reconstruct history from tickets, commit logs, and interviews.
The AI pilot that stalls at review. A text-to-SQL interface or an agent works in the demo and fails review, because the team can't produce the derivation behind an answer. The gap sits in the metadata underneath, which was never complete enough to defend an output. It stayed invisible until something started reading it.
Each of these is usually filed as a different problem: an operations problem, a project management problem, a compliance problem, an AI problem. They resolve to the same missing capability.
The Four Jobs
Documentation can do the first of these. A control has to do all four.
Job 1: Trace a published number back to the code that produced it
The trace has to run at column level and end at the actual expression, because that's where the answer is.
Passing looks like: pick a figure in a board deck or a regulatory filing, follow it back through every hop, and land on the specific line of code, the join condition, and the filter that produced it.
Table-level lineage answers "this table feeds that table," which is a different and considerably easier question. It gets you to the right neighborhood and leaves the last mile to a person with a SQL client and an afternoon.
Job 2: Predict what a change breaks before it deploys
A schema change is a proposal right up until it deploys. Every downstream consumer of the affected column is something that might break, and the list of them is knowable in advance.
Passing looks like: a proposed change goes in, a complete list of affected reports, models, pipelines, and applications comes out, and it happens before the change ships.
Most teams run this in reverse. The change deploys, something breaks, and lineage gets consulted during the incident. That's a forensics tool doing a planning job.
Job 3: Reproduce the estate's state on a date someone else picks
Regulators, auditors, and internal investigations all name dates, and they're rarely today's. The question is what a definition meant and where the data flowed as of the date they named.
Passing looks like: version history across the full scope of a model, with the differences from the current state visible side by side.
Current-state lineage answers today's question well. Any question about a closed quarter needs a time axis, and adding one after the fact means reconstructing history from tickets, commit logs, and the memory of whoever was there.
Job 4: Supply meaning to systems that consume metadata instead of reading documentation
Documentation assumes a reader who can ask a follow-up question. A metadata consumer answers with whatever it found.
Passing looks like: definitions, ownership, policy, and lineage available as structured metadata a downstream system can query, with the derivation attached to whatever it returns.
Five years ago this job barely existed. It's now the one growing fastest, and it's the one most existing lineage was never designed for.
The requirement is stricter than it looks. A wiki page describing a metric is readable by a person and invisible to an interface deciding between two columns with similar names. Anything a metadata consumer can't query, it will work around, usually by guessing.
Six Ways Lineage Quietly Fails
These failures are quiet by construction. Partial lineage looks exactly like complete lineage until the missing part is the part someone asks about.
Each of the six below is observable from inside your own organization, today, without a vendor conversation.
1. Coverage stops at a boundary
How you'd notice: a trace runs cleanly through four systems and goes quiet at the fifth. The gap is usually a mainframe, a homegrown integration, or a tool one business unit bought on its own.
What it costs: the chain breaks at the gap, so everything downstream of it is inference. Coverage gaps propagate forward through the whole trace.
2. Depth stops at the table
How you'd notice: you can see that a table feeds a report. You can't see which column produced which field, or what expression transformed it on the way.
What it costs: audit questions and incident investigations both operate at column level. Table-level lineage relocates the work to a human rather than answering it.
3. Freshness decays after the first collection
How you'd notice: ask when the lineage was last refreshed and you'll get an estimate. Ask what a full refresh costs and the conversation usually ends.
What it costs: unverified lineage gets checked by hand, which is precisely the cost the lineage was bought to remove.
4. Identity doesn't survive the crossing
How you'd notice: the same customer table exists in three systems under three names, and your lineage treats them as three unrelated objects.
What it costs: you never assemble a single view of how that entity actually gets used. Duplicate definitions multiply, because the original was invisible to whoever built the second one.
5. There's no time axis
How you'd notice: the tool shows the current state. There's no way to ask what it showed in March.
What it costs: every point-in-time question becomes a reconstruction project, staffed and scheduled, on a deadline set by someone outside the company.
6. There's no forward view
How you'd notice: impact analysis exists and gets used after incidents rather than before deployments.
What it costs: change velocity drops, because the safe move is to slow down and verify by hand. The tax lands on every release, including the ones that would have been fine.
What the Gap Actually Costs
Three costs. Only the third one shows up in a budget line.
The verification tax. Engineers who know their lineage has gaps but can't say where end up verifying by hand what a trustworthy system would settle. They build buffers into every estimate. Automation that should be possible stays unbuilt, because the inputs aren't trusted enough to act on without a human in the loop.
The human integration layer. In most large estates, one or two people hold the model no tool holds: which system's output to trust in which context, which gaps are real and which are artifacts, which changes cascade and which are safe. That's institutional risk sitting in two heads. When those people are unavailable, the estate goes opaque.
Reconciliation across an assembled solution. Most enterprises arrived at their current state by buying separately: a lineage tool, a catalog, a governance workflow tool, and a semantic layer product for the BI stack. Each keeps its own metadata store, connects to sources on its own schedule, and holds its own model of what a column is. Reconciling them is standing work that grows with every source added, and no amount of API maturity in any one tool retires it. Recent consolidation across the category adds to it: acquired products keep their original metadata stores for years, so a merger on a vendor's org chart doesn't merge anything in your estate.
Audit preparation is where all three surface at once. It runs as a project with a staffing plan, rather than as a query someone runs on a Tuesday.
A Diagnostic You Can Run This Week
Pick one number that matters: a figure in a board deck, a line in a regulatory filing, or a metric an executive watches weekly. Run these six questions against it using the tooling you already own.
Count the passes. Most estates land on two or three.
The pattern matters more than the score. Failures clustered in questions 1 and 2 point at a coverage and parsing problem, while failures clustered in 5 and 6 point at an architecture that only ever records the present.
Failures in 3 and 4 usually mean the metadata is spread across several stores that were never designed to agree.
Then run the same six against a second number from a different part of the estate, ideally one that touches a legacy system. The delta between the two scores tells you whether your lineage problem is uniform or concentrated.
A uniform problem is an architecture decision. A concentrated one might be a connector.
Write the answers down. Six months from now, in a vendor evaluation, this becomes the list of things you insist on seeing demonstrated against your own data rather than theirs.
What Comes Next
Two things follow from the score.
The first is architectural. Lineage gets derived in fundamentally different ways, and the derivation method decides which of the four jobs are reachable at all. Lineage parsed from code artifacts answers questions about what was designed. Lineage inferred from run-time logs answers questions about what happened to execute during the observation window. Those are different products with different ceilings, and the difference shows up on questions 1, 2, and 6 of the diagnostic above.
The second is operational, and it's the one most evaluations miss. Every failure in the six above is a decay problem. Coverage was complete once. Freshness was current once. The estate kept changing and the metadata didn't keep up, because the work of keeping up had no owner.
MITI calls the alternative MetadataOps: metadata management run as a continuous enterprise operation rather than a periodic project, with an owner, a lifecycle, and a quality measure. A team that fixes coverage without staffing the practice will be back at the same score in 18 months.
The MetadataOps Kit covers the practice, the operating lifecycle, a metadata quality scoring model, and a paste-ready job description and skills matrix for the MetadataOps Engineer role. It's available at www.metakarta.com/resources/metadataops.
Run the diagnostic first. Walking into an evaluation knowing which jobs you're failing changes the questions you ask, and it changes which demos you decline to sit through.
