Semantic Layer Decay: 95% to 65% in One Month
Analytics accuracy is a maintenance problem, not a model problem: Anthropic watched it drift from ~95% to ~65% in a month. Data ontology is now a battleground.
The accuracy of an enterprise analytics agent is not decided by which model you point at your warehouse. It is decided by whether anyone keeps the semantic layer current. In Anthropic's published write-up of its internal analytics stack, one number rarely gets quoted: when the skills package goes unmaintained, offline accuracy drifts from roughly 95% to roughly 65% within a month.
That same week, Constellation Research told enterprise buyers to treat the data ontology — the explicit definition of business concepts and the relationships between them — as a strategic battleground rather than a documentation exercise.
Both signals point the same way: a semantic layer and its ontology are not deliverables. They are assets that decay.
1. The shift is already in production
Start with scale. As of October 11, 2026, AIUseCaseHub had classified 1,114 of its 3,891 archived enterprise AI deployments as agentic, 420 of them multi-agent — and agents have grown from 7% of newly published deployments in Q1 2022 to 22% in Q3 2026, a 3.3x rise. Gartner's forecast is blunter: by 2028, one third of enterprise software will include agentic AI, up from less than 1% in 2025. A McKinsey survey of nearly 2,000 enterprise employees found 23% already scaling agents in at least some business processes, with another 39% in trials.
When the consumer of your data switches from a person to an agent, something quiet breaks. Analysts absorb ambiguity through context: they know that this revenue question means the tax-inclusive figure. An agent cannot know that. It picks the field that looks closest.
2. Why feeding it more data does not help
Anthropic's team frames analytics failure in three categories rather than "the SQL was written wrong":
| Failure mode | What it looks like | What you get |
|---|---|---|
| Concept-to-entity ambiguity | "Active users" maps to dozens of field-and-filter combinations; the agent cannot tell which one the asker meant | A plausible number with the wrong definition |
| Staleness | Sources, definitions and schemas change constantly; the agent's cached understanding rots within weeks | Conclusions that quietly drift |
| Retrieval failure | The right table exists and is annotated, but the search space is too large to find it | A correct-sounding answer from the wrong table |
The most counterintuitive evidence is an ablation. The team gave an agent full-text search across the entire corpus of dashboards, transformations and analyst notebooks — thousands of SQL files — and verified that it actually read them before answering. Accuracy moved by less than one point. Worse: for the questions it got wrong, the answer was present in that corpus about 80% of the time. Availability did not predict correctness.
Chinese analysts reviewing the same case reached an identical conclusion. In its 2026 Data+AI Application Practice Report, the Chinese research firm 爱分析 (iFenxi) notes that opening thousands of historical SQL files to an agent improved accuracy by less than one percentage point: volume of material cannot substitute for structured data navigation.
Our own field test against a real enterprise warehouse landed on the same rule: answer quality tracks whether the question resolved to the right entity. And a benchmark points the same direction. dbt Labs' 2026 test — 11 questions, 20 runs each, on the ACME Insurance dataset — found a double-digit gap between the two paths on the same modeled project:
| Model | Text-to-SQL | Semantic layer |
|---|---|---|
| claude-sonnet-4-6 | 90.0% | 98.2% |
| gpt-5.3-codex | 84.1% | 100.0% |
The report's own framing is worth keeping: when the semantic layer can answer a question, it gets it right; when it cannot, it says so. It never silently returns wrong data.
Structure beats access, and two independent bodies of evidence agree. Whether that structure survives the next quarter is a maintenance question.
3. Ontology lands on the vendor roadmap
In October 2026, Constellation Research made "semantic and context management" its own evaluation category (a 2026 ShortList), selecting 8 solutions from more than 20. Its advice to buyers deserves to be read literally: solving each new AI use case with custom semantics, custom retrieval and custom policy logic builds up a backlog of data debt and rework. The recommended move is a low-regret one — hold off on over-committing to definitions, relationships, policies and interfaces, and spend on portability while the market settles.
Catalog vendors are repricing themselves accordingly. At its revAlation conference, Alation shipped six products (AI Governance, Ontologies, Intelligent Feeds, Console, Governed Collections, Semantic Model Mastering). Analysts described the direction as a shift from passive catalog documentation for humans to an active, runtime enforcement system for enterprise data, context and AI agents. Two pieces speak directly to this article:
- Semantic Model Mastering does not build another semantic layer — it reconciles the ones you already have. It ingests semantic models from Databricks and Snowflake, governs and enriches the definitions inside Alation, then syncs them back, so that a concept like "active customer" resolves to one definition across downstream analytics and agents instead of each team redoing the governance.
- Ontologies turn business rules into explicit, reviewed artifacts. Relationships, constraints and process logic are defined and approved by domain experts, mapped to the catalog, and exported on open standards so different agent runtimes can consume them.
One more number from the same release: in Grant Thornton's 2026 survey of 950 C-suite executives, 78% were not confident their organization could pass an independent AI governance audit within 90 days.
The practical path is less dramatic than ripping out your data governance program. It is four additions: point core business concepts at authoritative sources; make data carry its business meaning and usage notes; expose data and knowledge to agents through governed, callable interfaces (semantic layer, metric platform, data service, knowledge base); and make every call traceable and updatable. Note that last clause — the requirement is not "built once", it is "traceable and revisable".
4. Treat the ontology as an asset: four moves
| Move | Failure mode it addresses | How you verify it |
|---|---|---|
| 1. One authoritative entry per business concept | Concept-to-entity ambiguity | Can you name the owner and scope of a given metric? |
| 2. Commit model changes and meaning changes together | Staleness | Does the data-model pull request also touch semantic or skill files? |
| 3. Re-run evals on every change, show sources online | Retrieval failure and silent failure | Can a conclusion carry its source, definition and freshness? |
| 4. Keep definitions portable | Lock-in and rework | Swap the agent runtime: does the same question return the same number? |
Move two has a measurable reference point. Anthropic folded semantic and skill maintenance into the data engineering workflow: a review hook flags any data-model change that does not also touch a skill file, and roughly 90% of data-model pull requests now include a skill change in the same diff. Maintenance survived because it lives on the same review pipeline as the table change.
Move three comes with an honest boundary. Their online monitoring keeps an adversarial review step (+6% accuracy, but +32% tokens and +72% latency) precisely because high accuracy is not the same as eliminating error: silent failures still happen, so conclusions escalated to management keep a human sign-off, while ordinary answers carry provenance — source trust tier, data freshness and the responsible owner.
Closing
The real change this week is not that a model got better. It is that an enterprise now has to assign an owner to meaning itself: who defines it, who updates it when the model changes, and who notices when it stops being true.
An ontology without a maintenance loop is just a well-written document, and documents do not stay correct on their own. In OntiCards' four-layer architecture this responsibility sits in the semantic card layer: every business concept is a data card that carries its meaning, query scope, sensitivity level and owner, so model changes and card changes travel the same review path and agents can only reach data inside the boundaries those cards define — which is why an ontology-backed knowledge graph holds up better than flooding the model with historical SQL. If you are running a natural-language analytics pilot and the definitions are getting messier as usage grows, write to hello@onticards.com and tell us about your setup.
