Same Answer, One Is Wrong: Text-to-SQL Must Be Checkable
Two AI queries return the same number; one is wrong. VeriPIE from KNOWIDEA and Georgia Tech says voting can't catch it — answers need checkable evidence.
The headline first: the most dangerous text-to-SQL error isn't the query that fails — it's the query that runs, returns a plausible number, and computes the wrong thing. On October 13, 2026, decision-intelligence company KNOWIDEA, together with researchers from Georgia Tech and BlackRock, published the VeriPIE paper, which puts the problem squarely on the table: two AI-generated queries against the same database and the same question can return identical answers today — and tomorrow, a single new record makes one silently drop accounts while the other reports them faithfully. No error message. No warning.
The direction the paper points to is clear: the competitive frontier for data Q&A systems is shifting from "how accurate" to "how checkable" — every answer should ship with evidence a risk team can independently validate before acting on it.
1. The classic trap: two ways to write "no matching payment"
The paper's most unsettling example comes from a perfectly ordinary business question: find the records with no matching payment.
There are two common ways to express that condition, and on current data, both queries return exactly the same result. Then a payment record with a missing account ID shows up — one version silently excludes every unmatched account, while the other still returns them. The report looks complete. It isn't.
KNOWIDEA's press release names the consequences directly: a wrong query "can silently omit exposures, double-count transactions, or miscalculate financial aggregates while returning plausible results."
- Risk: an omitted exposure understates a portfolio's risk.
- Reporting: double-counted transactions misstate a regulatory filing.
- Operations: a reconciliation that drops unmatched accounts hides the very breaks it was built to find.
This is not hypothetical. Our real-enterprise text-to-SQL reality check covered the same failure family back in August: a naive query missing a deduplication step inflated revenue by 76%, and the output looked perfectly reasonable until someone who understood the business reviewed it.
2. Why majority voting can't save you
The standard engineering response today is self-consistency voting: generate multiple SQL candidates and go with the answer most of them agree on.
The VeriPIE paper argues this is inadequate, and the reason is one sentence with real weight: agreement on one database does not establish semantic equivalence. Two queries can return identical results on today's data and diverge the moment the data changes. What voting selects is "everyone happened to agree today," not "the computation is correct."
The paper offers an even subtler example: replacing an explicitly supplied SUM(x)/COUNT(*) formula with AVG(x). The former counts every row in its denominator; the latter excludes nulls. On data with no null values the two are identical — introduce one NULL, and the results silently split.
3. VeriPIE's answer: make every candidate bring evidence to the interview
VeriPIE is neurosymbolic: a language model generates, a theorem prover checks. The pipeline looks like this:
- Generate structured query plans: instead of emitting SQL directly, the model first writes a plan in QuerySketch, an intermediate plan language derived from the Query Plan Language (QPL), spelling out which tables to scan, how to filter, join, and aggregate;
- Lean-backed checking: plans are parsed and type-checked against the schema (tables, columns, types, aggregation, grouping, output shape); rejected plans come back with structured diagnostics that drive model repair;
- Answer contracts: expected structure derived from the question becomes checkable rules, such as "the output must include these two columns";
- Equivalence buckets + bounded counterexample search: candidates with identical results are grouped, then an SMT-based search looks for a counterexample database within a fixed bound — if two queries differ on any legal data instance, the "false equivalence" is exposed on the spot;
- Selection: the final ranking combines execution consensus, plan checks, and counterexample evidence.
One detail that captures the design philosophy: the system ships a handwritten safeguard specifically to catch AVG(x) substituting for SUM(x)/COUNT(*) — syntactically legal, semantically swapped.
How well does it work? The paper reports single-run experiments on the complete Spider development set and on BIRD:
| Metric | VeriPIE | Baselines |
|---|---|---|
| Spider dev test-suite accuracy | 72.05% (745/1,034) | DeepEye 65.67% (679/1,034) |
| Spider two-query coverage | 78.43% (811 questions) | DIN-SQL 74.66%, DeepEye 69.05% |
| BIRD execution accuracy | 68.90% | — |
| BIRD two-query coverage | 73.73% | — |
On raw accuracy the gains are incremental. And the paper is refreshingly honest about the limits: these checks do not prove that a query matches the user's intent. Their value is exposing malformed plans and hidden disagreements — turning "why should I trust this answer" from a verbal assurance into a stack of evidence you can actually read.
4. What this means for enterprise deployments
The timing is no accident. The paper opens by anchoring itself in finance: model risk management (MRM) frameworks demand independent validation, transparency, and effective challenge; the Financial Stability Board lists AI-related model risk, data quality, and governance among financial stability vulnerabilities; the EU AI Act adds risk-based compliance requirements. In other words, in regulated settings, checkability isn't a nice-to-have — it's table stakes.
For enterprises rolling out data Q&A capabilities, we'd put three items on the agenda:
First, every answer carries an evidence chain. Question → metric definition → tables involved → query plan → check results. The chain must be complete, traceable, and human-readable. Users should receive "answer plus provenance," not a bare number.
Second, push definitions into the semantic layer. A business definition like "no matching payment" needs exactly one governed, authoritative home — not a private copy in every analyst's SQL. That's the same direction argued in this week's discussion of turning data into context: business definitions must become inputs to the query process. At OntiCards, metric definitions and relationships are captured as ontology assets, so agents align on definitions before a query is even written. See how this fits our four-layer architecture.
Third, make validation routine. Rather than waiting for an audit to find the problem, make "is this answer right" part of the daily loop: data validation and reconciliation should live inside the Q&A pipeline, comparing and surfacing disagreements automatically. It's the same principle we keep returning to in agent governance: trustworthy beats clever.
Models will keep getting stronger and benchmark scores will keep climbing. But as long as enterprises rely on these numbers for regulatory filings, risk control, and decisions, "looks right" will never be enough. Checkable is the passing grade for data answers.
