SQaLe

Towards Specialized Text-to-SQL Models with SQaLe

Abstract

Frontier language models in agentic pipelines lead text-to-SQL benchmarks, at a high inference cost. Small specialised models would avoid that cost, but training them needs data that reflects the scale, semantics and structure of real databases. SQaLe is a semi-synthetic text-to-SQL dataset built on 9,259 real-world schemas from SchemaPile. Our generation pipeline extends each schema, fills it with synthetic rows, generates questions and answers them with an agent, and validates every stage by execution. The result is 1.4 million natural-language questions paired with SQL, over schemas far larger than in any existing dataset. Training a 2B parameter model on SQaLe with GRPO alone raises its execution accuracy on an independent test set from 38.7% to 66.3%. It does so by learning effectively how to explore the dataset. It finds the relevant tables in large schemas and reads the values its answer needs.

Why SQaLe

Small specialised models avoid the inference cost of frontier agentic pipelines. Training them needs data that looks like the databases they will be used on, and the common text-to-SQL training sets use small schemas. Yet, BIRD’s training set has significantly fewer samples, while SynSQL lacks realistic schemas. A model trained on them does not learn to search for the tables a question needs effectively.

SQaLe starts from 9,259 real-world schemas from SchemaPile, extends each one with tables in its own style and fills it with synthetic rows. Its schemas have a median of 113 tables and 538 columns, while also delivering enough questions and SQL queries to make for an effective training dataset.

The dataset

question–SQL pairs
1.4M
real-world schemas
9,259
distinct SQL queries
176,761
median tables per schema
113
median columns per schema
538
synthetic rows
109M

Each record pairs a natural-language question with a schema and a gold SQL query that has been run against the populated database. Every question exists in 8 phrasings, the original and seven rewrites in different styles, and all eight share one SQL query. One record is laid out in full on its own page, with the schema it was written against.

The dataset is published on Hugging Face in two parts that join on schema_id. trl-lab/SQaLe-2-text-to-SQL-Queries holds the questions in all eight phrasings, the gold SQL and its result. trl-lab/SQaLe-2-text-to-SQL-Schemas holds the DDL and the generated rows of each database.

The result

To isolate the effect of the data, each of the three runs starts from the same Qwen3.5-2B checkpoint and trains with GRPO on one corpus: SQaLe, SynSQL-2.5M or BIRD train (MSQaLe, MSynSQL and MBIRD). MSQaLe answers as well as untrained models four times its size. It sits 28 points above the best untrained model of its own size, 52% against 24%, and it takes an 8B general-purpose model to reach the same accuracy. Qwen3.5-27B leads at 63%.

Accuracy against model size, 100 moderate SQaLe test questions

All 25 models as a table
Figure 1. Accuracy against parameter count for every model in the same agentic harness, with dot area proportional to TFLOPs per question. The grey step line is the best accuracy an untrained general-purpose model reaches at each size or below, the bar at 2B is MSQaLe’s lead over it, and the dashed line runs to the smallest untrained model that matches it. Hover or focus a dot for its values; arrow keys move between dots.

Training on SQaLe

Apart from the training data and the strength of a length curriculum, the base model, environment, tools, reward, steps, batch size and evaluation are identical. There are no distilled traces, no teacher and no test-time scaffolding.2

On the SQaLe test set (Figure 2), MSQaLe reaches 66.3%, sitting far above the base model and also outperforming MBIRD and MSynSQL by considerable margins.

Execution accuracy (%) by training corpus

Figure 2. Execution accuracy of Qwen3.5-2B trained with GRPO on one corpus each, on 300 SQaLe test questions, BIRD and EHRSQL. Bold marks the best of the three trained models in each column.

What a small model learns

The three trained models differ only in their training data, so differences in how they behave come from the data. We look at four of those differences.

Exploration

The environment caps rounds, not tool calls, so the only way for the model to see more of a database is to ask more in each round. Over training it learns to do that: tool calls per episode roughly triple, while the number of rounds stays near six. Three real episodes are replayed round by round, on schemas of 106 to 239 tables.

Inside one episode, over GRPO training on SQaLe

Figure 3. A round is one reply from the model: it writes a short plan and issues as many tool calls as it likes, and the environment runs all of them and returns the results in one message. The model is allowed six rounds whatever it does, so the rising line is more of the database read within the same budget. Data: episode shape (CSV), run 26640986, which reported up to step 1,658 of 1,800.

Schema size

Accuracy falls for every model as the schema grows, and MSQaLe leads at every size, from 72.7% with only the gold tables to 66.3% on the full schema. With only the gold tables present, all three models open every gold table in at least 95% of episodes. On the full schema, MSQaLe opens every gold table in 89.7% of episodes, again outperforming MBIRD and MSynSQL. BIRD and SynSQL training schemas have a median of 5 and 10 tables, so models trained on them never have to search.

Populated tables

Before submitting, MSQaLe has seen 88% of the string literals its answer depends on. MSynSQL has seen 55%, MBIRD 40% and the untrained model 35%. SynSQL’s tables hold a median of 2 rows, and SQaLe’s hold 69.

Schema size

Share of episodes in which the model opened every gold table (%) at three schema sizes

All nine values as a table

String literals seen before submitting

Share of the string literals the answer depends on that the model saw in tool output (%)

Figure 4. Top, the three trained models at three schema sizes: the gold tables only, the gold tables plus 32 distractor tables, and the full schema. The toggle switches between accuracy and the share of episodes in which the model opened every gold table. Bottom, how many of the literals its answer needs each model has seen before it submits.

Domains

On SQaLe test questions whose schemas lie outside BIRD’s domains, MSQaLe leads MBIRD by 9.9 points. Inside BIRD’s domains the lead is 2.8 points.3

How it compares

How much searching a model has to learn depends on how large its training schemas are. Figure 5 compares schema size across four datasets. The median SQaLe schema has 113 tables and 538 columns. The next largest medians are EHRSQL’s 13.5 tables and 92 columns, measured on 2 schemas. SQaLe also has the most foreign-key relations, 1,196,078 across its 9,259 schemas.

Figure 5. Median tables (open circle) and median columns (filled circle) per schema, on a log scale. The table below gives the full schema statistics; EHRSQL does not report rows per table.

23.4% of SQaLe queries are nested, against 7.7% in BIRD, and 40% of the queries that join chain multiple joins, against 26% in BIRD. 3.1% of queries touch five or more tables, up to 20. No BIRD or EHRSQL query touches more than four.

How it is built

Schemas that size do not come ready to use. The pipeline runs in five stages, and every stage is validated by execution. Value synthesis repairs rows that break key constraints, and answering sends failed or rejected queries back to the agent.

Show how it is builtHide how it is built
Figure 6. The generation pipeline. Return arrows mark the repair loop in value synthesis and the two feedback loops in answering.
  1. Schema collection and extension. SchemaPile provides real-world schemas from permissively licensed GitHub repositories. A tool-using agent annotates each of the 14,597 source repositories with a domain description. An LLM then extends each schema with tables that keep its naming conventions, normalisation level and foreign-key style.
  2. Value synthesis. Tables are filled in foreign-key dependency order. For each table the LLM writes a Python function from its DDL, its original rows, the allowed foreign-key values and the domain description. Fact and junction tables get more rows and a skewed key distribution, and every table is checked for primary-key uniqueness and referential integrity.
  3. Question generation. A connected subgraph of up to 20 tables is sampled along foreign keys, together with sample rows. The LLM writes questions that need every table in it, at three difficulty levels (simple, moderate, hard), with every literal grounded in the data.
  4. Style variation. Each question is rewritten into the seven styles shown in the example record. All eight versions share the gold SQL.
  5. Agentic answering and judging. An agent explores the live database with tools (list_tables, describe_table, sample_rows, distinct_values, run_query) and commits with submit_sql. Execution errors go back to the agent. An LLM judge checks the result against the question and sends rejected queries back for a rewrite.1

Cite

If you use SQaLe, please cite the paper, Towards Specialized Text-to-SQL Models with SQaLe (PDF), and the earlier workshop paper.

BibTeX

Notes

  1. On 158 BIRD dev questions (225 judge calls), 94.5% of the queries the judge accepts are aligned with the question. The judge agrees with human labels 86.2% of the time (κ = 0.68).↩
  2. The schema is withheld. The model explores the database through tools, the five used in answering plus foreign_keys and join_path. The reward is tiered (correct result, then executes, then parses, then nothing), and partial credit never ranks a wrong query above a correct one.↩
  3. The domain split covers 108 schemas. That sample is too small to call the difference between the two gaps significant.↩