Writing ·
The business risk of a confidently wrong answer from an LLM
Text-to-SQL works beautifully until the schema is real ends its central point in a single sentence: nobody notices until someone reconciles the number against a report built by a person, months later. As Austin Powers would say, “that train has sailed”.
This is about those months. The question is not why the answer was wrong, that piece covers it, but what a wrong answer costs once it has left the database. That is a business risk rather than an engineering one, it is much harder to bound, and it is discussed far less than the engineering risk that usually stands in for it.
Is the business risk any different from Eventual Consistency?
A lot of modern distributed databases trade precision for performance by using Eventual Consistency, where not all nodes will be aware of a recent update for an appreciable time afterwards. The business risk of an LLM using the wrong data to answer is in a completely different category of risk. Eventual Consistency means that within a few seconds or even a few hundred milliseconds there will be one (and only one!) answer to a question, and it will be the correct one. When text to SQL goes wrong we can’t even rely on the same wrong answer being produced: without a clear plan or single structured SQL statement the LLM could return different wrong data each time it’s called.
A number does not stay where it was produced
The wrong answer will carry the authority of 4 digits of precision, and be delivered with the breezy confidence of an LLM.
Then somebody pastes it into a slide. The slide goes into a pack. Somebody quotes the pack in a meeting, and by then the number has acquired a property it never earned: it is now the figure, repeated by a person rather than produced by a tool. A week later it is in a forecast. A month later it is the baseline that next quarter’s number is compared against. Then it’s in the 10Q.
Every hop strips provenance. At no point does anyone do anything wrong. Nobody is careless. The number simply travels faster than its caveats, which is what numbers do, and has done since long before anyone connected a model to a database.
What is new is the scale at which this can start happening. A person producing that figure would have had to write the query, and writing the query is where the intra-company transfers that should be excluded get noticed. There’s a reason most enterprises have an entire department of database people whose job is to interpret the numbers.
Why it is structurally hard to catch
Ordinary defects announce themselves. This one has three properties that suppress detection, and they compound.
There is no error. Nothing failed. There is no stack trace, no alert, no red entry in a log, because from the database’s point of view a perfectly valid query returned a perfectly valid result.
The only detector is reconciliation, and it is periodic. Somebody comparing two independently produced numbers is what finds this, and that happens at month end, at audit, at year end, or when a customer queries an invoice. And we’re assuming that AI wasn’t used for both numbers. The gap between production and detection is therefore not a function of how wrong the number was. It is a function of your reporting calendar and how much ‘old school’ DBA skills remain in the business, and can spot when an LLM used the wrong data to answer a question.
The query may no longer exist. If the model composed SQL on the fly, the text that produced the figure was never written down anywhere a person would look. You cannot re-run it, diff it, or ask its author what it meant to count. This is the part that turns a wrong number into an unanswerable question, and it is worth knowing that your database can tell you whether it is happening — detecting ad-hoc SQL in Oracle is two queries against Oracle’s own views.
The cost is a function of time to detection, and it is not linear
There is a long established principle that the cost of a bug in software increases by a factor of 10 at each stage of the design and deployment cycle. This applies here.
Caught in the session. Someone who knows the data looks at the figure and says “that’s too high”. Cost: a few minutes, and a small amount of trust in the tool. This is the case every demo implicitly assumes, because in a demo there is always somebody watching who knows the answer.
Caught at reconciliation. Weeks or months later, two numbers disagree. Now the cost is the investigation — which figures were derived from the bad one, who saw them, what was decided — and the investigation is expensive precisely because provenance was stripped on the way. You are reconstructing a chain that nobody recorded.
Caught outside. A customer, an auditor or a regulator finds it first. This is where the business risk of connecting an LLM to real data actually sits, and it is nowhere near the model. At this point the number itself is the smallest part of the cost. What you are now managing is a question about control: how did this figure come to be produced, who approved the thing that produced it, and what else did it produce. That question is not answerable by fixing the query.
The reason this matters for how you build is that the first regime is the only cheap one, and it is the only one that depends on a human being present. Every unattended deployment moves the expected cost from the first regime to the second or third.
This is not an ordinary bug, and testing does not reach it
Software engineering has good defences against code that is wrong. Tests encode what the answer should be; a failing test names the defect. None of that machinery applies here, because nothing in the system knows what the right answer was. The whole reason to ask the question was that nobody already had the number.
A test can tell you the tool returned a number. It cannot tell you the number was the one you asked for. That is not a gap in your test suite — it is the shape of the problem, and it is why “we’ll catch it in QA” is not a plan.
What actually reduces the exposure
Not model accuracy, because accuracy is not observable at the moment of use. Three things that are.
The number comes from a query a person wrote and can re-run. Then the answer has an author, and “what does this count?” has an answer that does not require archaeology.
You can tell afterwards which tool produced which figure, with the values it was given. A count of calls does not help six weeks later; the parameters do. This is what an audit trail is for, and the reason to record values rather than volumes.
The set of answers the system can produce is enumerable. If the operations are a curated list, then the question “what could this thing have told someone?” is answerable by reading the list — which is also the argument in what “the agent cannot compose SQL” means on a real schema, stated there as a claim about capability and here as a claim about recovery.
None of these makes an answer correct. They make a wrong answer findable, and given that detection time is the dominant term in the cost, findable is most of the battle.
The honest limit
Curated tools do not make your numbers right.
They make the agent’s number the same number your reporting layer gives, which is a smaller and more defensible claim. If the query in your reporting layer has been quietly wrong since 2019, an agent calling it is now confidently wrong at machine speed, and nothing described above helps you.
What you have bought is that the error is one error, in one place, with an author, that a person can find and fix — rather than an unbounded set of one-off errors that were never written down. That is a real improvement and it is worth being precise about its size.
How is MCP DB Wizard relevant to this?
If you arrived here from a search, the short version first. MCP DB Wizard is a generator, not a gateway you put in front of your database. You point it at an Oracle schema, tick the tables, PL/SQL routines and tested SQL statements one job needs, and it writes a working MCP server exposing those and nothing else, which you run as a Docker image. It is relevant to this article because each of the three things above stops being a practice somebody has to keep up and becomes a property of the artefact.
The number came from a query somebody wrote. There is no run-sql tool — none is generated — so
the agent has no way to compose a statement of its own. Every figure it returns came from a PL/SQL
routine your team already maintains, or a SQL statement somebody wrote, tested and put in the
config. “What does this actually count?” therefore has an author, and asking again
next month returns the same number rather than a new one.
The provenance outlives the answer. The audit trail records the values a tool
was called with rather than a count of calls, which is the difference between reconstructing a chain
six weeks later and reading it. The database keeps its own copy of the same fact: every statement
carries a /* Created By MCPDBWizard */ comment and the connection sets its MODULE, so a figure
can be attributed from Oracle’s own views without taking our word for anything. That is what
detecting ad-hoc SQL in Oracle is for, and it works
against any server, including ours.
The set of possible answers is a file. The operations the server exposes are the config, and the config is reviewable text — read before deployment, diffed afterwards. So “what could this thing have told somebody?” is a lookup rather than an investigation, which matters most in the third regime, when the person asking does not work for you.
What it does not do is put a person back in the room. The first regime — someone who knows the data saying “that’s too high” — is the cheap one, and nothing here recreates it. What changes is the expected time to detection, and since that is the term the cost depends on, it is the term worth attacking. The limit in the section above still stands: a query that was already wrong is still wrong, and now it is fast.
How do I get MCP DB Wizard?
It’s available from our GitHub repo as a Docker image. It supports Oracle 12c through 26ai, and is regression-tested against six live instances spanning that range.