On the seventh of August I built a company that doesn’t exist. Caldergate Distribution Group: a £260m UK distributor with three segments, a term loan with covenants, and eighteen months of finance reporting. 34 files, exactly as a real team would keep them. Thirteen Excel workbooks, seventeen PDFs, two board decks, two CSV extracts. I also buried three problems in it on purpose: an April report that was reissued four days later with £620k of spot revenue restated, a legacy tracker with a stale plan-basis footnote, and a £285k gap between a department export and the issued pack that only one file can explain.
Then I wrote 25 questions about it, with a cell-level answer key and a grader that runs as a script. The whole thing is public, answer key included. It’s called the Board Pack Test, and you can run it against any AI tool you’re being sold, including mine.
The design decision that matters
Each question is worth four points, 100 in total. Here’s the part I care about: 81 of those points can be checked by a script, with no human judgment anywhere. Did the answer state the right figure, and cite the document it actually lives in, down to the sheet and cell? Did it refuse the five questions whose answers aren’t in the pack, without smuggling numbers into the refusal?
The other 19 points need a human marker, because they’re about reasoning: did the answer show its working, did it explain why the reissued report supersedes the original. A script can’t grade that. But the split is the whole point of the exam. The 81 measures what you can trust without checking. The 19 measures what you’d have to check anyway.
What the frontier models did
I ran the raw models first, through their own APIs with their own file tools. Claude Opus 5 and GPT-5.5, the same 25 questions, three independent runs each, identical prompt wording and citation format. Claude cost me about $22.50 for its three passes. GPT, $6.50.
Their answers were mostly right. On the answered questions the raw Claude runs and mine land within a point of each other, and GPT isn’t far behind. All three traps got caught by everyone, every run. These models are genuinely good at reading finance documents, and anyone telling you otherwise is selling something.
The machine-verifiable score told a different story. Claude banked 66 of the 81. GPT banked 71. The missing points weren’t wrong answers. They were right answers that couldn’t be proven right by a script. Refusals that helpfully mentioned figures from adjacent periods, which the grader zeroes because it can’t tell honest context from a smuggled answer. Citations that pointed at a whole sheet, or a range, or in one case the correct number on the wrong sheet entirely.
The asterisk I want on my own benchmark
If a human grades everything and reads every refusal charitably, raw Claude scores 100. My own tool scores 99. I put that table in the repo and I’d rather you saw it here first.
With a marker holding the answer key, every frontier tool lands between 95 and 100. The public question set stops discriminating. That’s not a flaw I’m hiding. That’s the finding. The models are equally smart. What separates them is whether their correctness survives without the marker, because at your desk there is no marker. Nobody re-grades the AI’s answer against a key before it goes in the board pack. The checking is the work, and whoever does the checking is the product.
What I changed after losing the first round
The first version of my own tool, Grounded, banked 66 and 68 on the same exam. Roughly raw-model territory. The transcripts showed me exactly where the points went, so over one day I made three changes.
Refusals now carry no figures at all. If the answer is “that number isn’t in your documents”, it says so without quoting the numbers that are. Second, when two files carry the same figure, the tool now prefers the authoritative one: the reissued FINAL over the superseded original, the current pack over the legacy tracker. Third, citations now name the exact cell the claimed figure lives in, not a range that happens to contain it.
Re-run, three passes: 81, 81, 81 on Claude. 81, 79, 81 on GPT. Every machine-checkable point, banked, on the same underlying models. The guard behind this has a transcript I keep going back to: in one earlier run it drafted an answer, found three figures in the draft it couldn’t trace to any document or to a calculation it could show, and held the whole answer back rather than state them. It cost points on the day. That transcript is in the repo too, and I’m fonder of it than of the 81s.
The part the benchmark can’t show you
The raw models took this exam under lab conditions: every question got the entire corpus attached. At 34 files that was already creaking. One vendor’s code sandbox caps at 16 file uploads, so the harness had to paste the headcount CSV into the prompt as text, for every vendor, to keep the test fair. A real finance drive is hundreds of files and grows every month-end. Attach-everything doesn’t scale past the lab, and the moment you bolt retrieval onto a raw model to cope, you’ve started building the thing you were told you didn’t need. The proving layer is the hard part, and it’s the part that was worth 15 points here.
What I’d tell your CFO: the models are equally smart, and only one of the setups can prove it. A right answer you can’t verify still costs you the verification. Buy the proof, enjoy the intelligence.
Run it yourself
Don’t take any of this from me. The repo has the 34 documents, the 25 questions, the answer key, the grader, every transcript from every run, and the adjudication record, including the rulings that went against my own tool and the one that restored points to GPT. Clone it, run the grader, read the transcripts. Dispute a ruling by opening an issue; the dispute path is in the methodology.
Better still, ignore my fictitious company and bring your own pack. Ask your own 25 questions, the ones your team already knows the answers to. That’s how I onboard pilots, and it’s a better exam than mine, because it’s yours.