Writing ·
Text-to-SQL works beautifully until the schema is real
Everyone has seen the demo. Someone types “show me last quarter’s revenue by region”, a model writes some SQL, a table appears, and the room nods. It genuinely is impressive.
Then you point it at your actual database and it stops being impressive, in ways that are worth being specific about.
Benchmarks are tidy. Your schema is not.
Text-to-SQL benchmarks use schemas that were built to be understood: sensible table names, columns that mean what they say, a manageable number of tables, and questions that have one correct answer derivable from the structure alone.
Production schemas are not like that, and the gap is not a matter of degree. Three specific things break:
Scale. Hundreds of tables, sometimes thousands. The model cannot see them all at once, so something has to choose which fraction it looks at — and that choosing step is now part of your accuracy problem, before a single line of SQL is written.
Abstraction. Look at your production system and count how many tables refer to a real-world thing you could point at. Not many. The rest are abstractions that made sense to whoever designed them, and understanding them takes more knowledge than fits in a column name.
Exceptions. Every long-lived system has them. A BOOKINGS table that is mostly sales but also
carries intra-company transfers for historical reasons. A status code that means one thing before
2019 and another after. A “deleted” flag that three teams interpret differently. Getting the real
number requires a specific, complicated query that encodes knowledge no schema metadata contains.
The failure mode is the dangerous one
If a generated query failed loudly, this would be a much smaller problem. Mostly it doesn’t. It returns a number.
The number is plausible. It is formatted correctly. It is in the right ballpark, because it is counting something — just not quite the thing you asked about, because it counted the intra-company transfers too. Nobody notices until someone reconciles it against a report built by a person, months later.
Wrong-and-confident is worse than broken, and it is the normal outcome rather than the rare one. What those months actually cost — and why the cost depends on how long the number goes unnoticed rather than on how wrong it was — is the business risk of a confidently wrong answer.
What actually fixes it
Not a better model, and not a bigger context window. The knowledge that makes the query correct is not in the schema, so no amount of reading the schema recovers it.
The fix is to write the query once, correctly, with someone who knows about the intra-company transfers — and then let the agent call that, by name, with typed parameters. You already have these. They are the queries in your reporting layer and the PL/SQL your team has maintained for years, and they encode a decade of exactly the knowledge a model cannot infer.
Turning them into tools is a much smaller problem than teaching a model your business.
And the resource question
One more thing, less discussed. Real systems react badly to badly written or overly broad queries. A generated query does not just risk a wrong answer — it can burn serious resources, and on some production systems it can cause real trouble. A fixed statement with bind variables has a plan you have seen before.
How do I get MCP DB Wizard?
It’s available from our GitHub repo as a Docker image.