SayQL: Natural Language to SQL with GPT-Neo

Plenty of people who depend on a database can't get at their own data because they don't write SQL. SayQL fine-tunes an open-source language model to close that gap, turning a plain English question into a SQL query. GPT-Neo 1.3B is trained with a causal language-modeling objective on 7,000 question/SQL pairs from the Spider dataset, split 80/10/10 into train, validation, and test. Each example becomes a single string (the question, then the query, then a STOP marker) so the model learns to write the query after the question and stop on its own. Early stopping watches the validation set, and every reported number comes from the test split, which is never touched during training. Training ran 13 of a possible 50 epochs before early stopping, restoring the best checkpoint at epoch 3 (validation loss 0.944). On the held-out test split the restored model measured loss 0.935 and perplexity 2.55. Against the un-tuned baseline over 50 random test questions, BLEU rose from 0.002 to 0.455, exact-match accuracy from 0% to 20%, and valid-SQL rate from 10% to 90%. Simple queries come out nearly exact. Complex joins fail in one specific, explainable way: the model invents table and column names, which follows directly from the fact that it never sees a database schema, only the question.

Dataset

Spider (7,000 question/SQL pairs, split 80/10/10 into train/validation/test)

Models

GPT-Neo 1.3B fine-tuned with a causal LM objective via the HuggingFace Trainer, fp16 on an A100, early stopping on validation loss with a patience of 10 epochs

Evaluation

Test-set loss and perplexity, smoothed sentence-level BLEU, exact-match accuracy, valid-SQL rate via a sqlglot parse check

Results & insight

Valid-SQL rate 10% to 90%, BLEU 0.002 to 0.455, exact match 0% to 20%, test perplexity 2.55. Remaining failures trace to missing schema context rather than broken syntax.