How we measure
Proof, not adjectives
Every claim on this site has a number behind it. This page says where those numbers come from — and how Qry shows its work to you every day.
Head to head, October 2026
31 / 31
everyday questions
The built-in assistant of a leading data platform: 29 of 31 — two never finished.
5 / 5
hard questions
The same assistant: 1 of 5. The hard ones need statistics, partial years and values stored as codes.
8 s
median time to answer
Against 14 s for the built-in assistant, on the same warehouse.
The setup
Same questions, same data, same rules
- Data
- Two public sample datasets: TPC-H and a retail sample with documented tables.
- Questions
- 31 everyday questions a business user would ask, and 5 hard ones written to trip a careless analyst.
- Setup
- The same warehouse for both, and no instructions, skills or hand-tuning in either.
- Judged on
- Whether the answer was right — checked against the ground truth computed separately.
- Models
- Qry with Claude on Google Vertex AI. Repeated with GPT on Azure AI Foundry in the EU: also 31 of 31 and 5 of 5.
Why it differs
The difference is method, not model
Where the other assistant failed, it had guessed: it filtered on a value the data does not contain, read a trend into a year that was only half there, or skipped the statistics a question asked for. Qry looks at the data before it trusts its own assumption, and moves to Python when SQL is not the right tool.
- The same results with two different model families
- Every answer keeps the query it ran, so a wrong one can be found
Verified every day
What Qry checks for you, without being asked
Every answer shows its work
The SQL it ran, the tables it read and the passages it quoted are one click away — the answer can be audited, not just believed.
Figures that are not in the data are flagged
When a number in an answer does not appear in what Qry read, it says so next to the text, instead of letting it pass.
Migrations are proven, object by object
Forge runs tables, views, functions and procedures on both sides and compares the results before calling anything migrated.
Rules that go stale are caught
When a business rule names a table or column that no longer exists, the rule is flagged and its owner is told — before it misleads an answer.
Verified before it ships
Every change is measured before it reaches you
A new model, a new prompt or a new rule can make answers better — or quietly worse. We run banks of real questions through the real product, under real permissions, before a change ships, and each run records what was in force so a change in score can be traced to its cause.
- Run through the product itself, not a test harness that skips it
- Scores compared run to run, case by case
What we do not claim
- Language models are not deterministic: the same question can take a different path. That is why we measure banks of questions, not single answers.
- The head-to-head is one benchmark on public data. Your data, your questions and your rules will tell you more — which is why we offer a session on them.
See it on your own data
We'll set up Qry against a dataset you choose and walk you through it — one hour, no slides.