Text-to-SQL and AI Data Agents 2026: BIRD and Spider 2.0 Benchmarks for Claude, GPT and Open Models

"Show me revenue by region for the last quarter, without the returns." Every company wants managers to ask the database questions like this in plain language, and text-to-SQL is one of the most requested AI use cases we see in Croatia and the DACH region. The public benchmarks now give a clear answer to how far it works: the best system on the BIRD benchmark gets 83.4% of questions right, human data engineers 93%, and on realistic enterprise tasks the leading agents went from 17% to 97% in two years. This article reads the leaderboards as of 29 October 2026 and turns them into decisions.

The benchmarks that matter: BIRD and Spider 2.0

Older tests such as Spider 1.0 used small, clean schemas and are effectively solved. Two benchmark families now decide how the industry measures text-to-SQL. BIRD (Hong Kong University and Alibaba, 2023) uses 95 real databases with dirty values and asks for "execution accuracy": the generated query must return exactly the right rows. Spider 2.0 (ICLR 2025) goes further with 632 enterprise workflows on Snowflake, BigQuery and dbt projects, with schemas of hundreds or thousands of columns, nested data and multi-step transformations.

BIRD: from 40% to 83% in three years, still 10 points behind humans

When BIRD launched in 2023, ChatGPT reached 40.08% and GPT-4 54.89% on the test set. By September 2025 Ant Group's Agentar-Scale-SQL scored 81.67%, and on 22 October 2026 AT&T's AskData took first place with 83.40%. Progress has slowed: the top has moved under two points in thirteen months, while data engineers and database students score 92.96%.

The top ten is dominated by companies that run text-to-SQL on their own data platforms: telecoms (AT&T, China Telecom), Chinese tech groups (Huawei, Tencent, Ant Group, Xiaomi) and banks (Sber; JPMorganChase in the single-model track). Most entries are not a single model but a pipeline: schema retrieval, several candidate queries, execution and a selector that picks the answer.

Spider 2.0: enterprise SQL went from 17% to 97%

Spider 2.0 is closer to a real data warehouse. At launch in late 2024 the best system, o1-preview, solved 17.1% of the tasks and GPT-4o 10.1%. Classic academic text-to-SQL methods collapsed completely: DIN-SQL with GPT-4o scored 0.00% on Spider 2.0-Snow, CHESS 1.28%. Two years later, agents that explore the warehouse, run test queries and repair their own SQL lead every track.

The agent matters more than the model

The leaderboards allow a rare controlled comparison. With the same open Spider-Agent framework, newer models do better: GPT-4o 12.98%, Claude 3.5 Sonnet 15.54%, Claude 3.7 Sonnet 24.50%, Claude 4 Sonnet 25.78% on Spider 2.0-Snow.

But the same model in a better system moves much further. Claude 4 Sonnet scores 25.78% in Spider-Agent, 38.39% in Chat2DB-Agent and 61.43% in WindAgent. Claude Sonnet 4.5 goes from 79.89% (LingXi Agent) to 90.49% (Paytm's Prism Swarm with Deepthink). For GPT-4o the spread runs from 0.00% to 12.98%. The engineering around the model, meaning schema linking, exploration queries, self-correction and answer voting, is worth 2–3× the score.

Claude, GPT, Mistral and open models on the leaderboards

Anthropic models are the most frequent named models in the top systems. On Spider 2.0-Snow, Claude Sonnet 4.5 powers three entries above 79% and QUVI-3 with Claude Opus 4.6 reaches 86.28%. On Spider 2.0-Lite the best named-model entries use Claude Opus 4.5 (QUVI-2.3, 65.81%), and a plain Claude Code agent with Sonnet reaches 61.2%. On BIRD mini-dev the top entry, schemagate, runs on Claude Opus 5.5.

OpenAI models lead where code agents edit dbt projects: the best named-model DBT entries use GPT-5 (41.18%) and GPT-5.4 (39.71%), and o3 reaches 62.89% on Snow with ReFoRCE. Open-weight models are competitive: GLM 5.2 powers the Lite leader, gpt-oss-120b scores 65.45% on Snow and DeepSeek V3.2 70.38%. No Mistral model appears on the Spider 2.0 leaderboards, so for Mistral (including the EU-hosted option many of our clients ask about) you need your own evaluation.

Your SQL dialect costs accuracy

Most research targets SQLite, but companies run PostgreSQL, MySQL, SQL Server or Oracle. BIRD mini-dev measures the same 500 questions in three dialects. The leading system, schemagate with Claude Opus 5.5, scores 81.0% in SQLite but 72.4% in MySQL and 72.6% in PostgreSQL. The second entry, Ontology2SQL, drops from 70.2% to 65.8%. Expect 5–9 points less on your production database than on a published SQLite score.

From benchmark to production: a checklist

Your own databases are rarely as well documented as a benchmark. In a 2023 study by data.world on an insurance data model, GPT-4 answered 16.7% of business questions correctly on the raw SQL schema and 54.2% when the same data was described by a knowledge graph. Context beats model size, which is why a semantic layer is the first step, not an extra. (More on this in our article on context engineering.)

• Build a semantic layer first. Business names, metric definitions ("revenue" = net of returns?), join paths and example queries. This is where most of the accuracy comes from.
• Use an agent, not a single prompt. Let the system look at sample rows, run test queries and repair errors, as the top Spider 2.0 entries do. Tools like the Model Context Protocol connect Claude to databases in a controlled way.
• Read-only and limited. A dedicated database role with read rights on approved views only, row limits, statement timeouts and a cost check (EXPLAIN) before execution.
• Show the SQL and the assumptions. Users should see which tables, filters and definitions were used, and the system should ask back when a question is ambiguous.
• Build your own evaluation set. Take 100–200 real questions with verified answers from your query logs and BI reports, and measure every model or prompt change against them. See our guide to AI evaluation.
• Mind GDPR. Personal data in query results goes to the model provider. Use views that mask personal fields and an EU data processing setup (GDPR and generative AI).

What this means for companies in Croatia and DACH

For a mid-sized company with an ERP, a CRM and a few reporting databases, text-to-SQL is realistic today for well-defined reporting questions on curated views, at an accuracy that the benchmarks put in the 70–80% range before your own tuning. It is not yet a replacement for the analyst on open, ambiguous questions. Croatian and German column names and questions add another layer of difficulty (see our measurement of multilingual model performance), so the semantic layer should map local terms explicitly. A sensible start is one department, 20 core metrics and an evaluation set; expand once the measured accuracy on your own questions is above 90%.

Frequently asked questions

How accurate is text-to-SQL in 2026?

On the BIRD benchmark the best system answers 83.4% of questions correctly (October 2026), compared with 92.96% for human data engineers. On enterprise Spider 2.0 tasks the best scores are 96.7% on Snowflake, 76.2% on the harder Lite setting and 65.6% on dbt projects. On your own data, accuracy depends mostly on schema documentation and evaluation.

Which AI model is best for text-to-SQL?

The best results come from agent systems, not single models. Claude Sonnet 4.5 and Opus models power several top entries on Spider 2.0 and BIRD mini-dev, Gemini 3 Pro powers the best named-model Snow entry, GPT-5 leads among named models on dbt tasks, and open models such as GLM 5.2 and DeepSeek V3.2 are competitive. The same model can score 2–3× higher in a better agent framework.

What is the BIRD benchmark?

BIRD is a text-to-SQL benchmark from 2023 with 95 real databases and dirty values. It measures execution accuracy: whether the generated SQL returns exactly the correct result. Human experts score 92.96%; the leaderboard top was 83.40% on 29 October 2026.

Is it safe to let an LLM query our production database?

Only with guardrails: a read-only database role restricted to approved views, row limits, timeouts and cost checks, masking of personal data, logging of every query, and showing the generated SQL to the user. Many companies start on a reporting replica rather than the transactional database.