Eval history
Two evals measure the real pipeline. Extraction is checked against as-reported SEC XBRL figures. Q&A is checked for groundedness: every figure an answer states must appear in the excerpts it was given. Every push re-scores each run below under the current rules.
Extraction accuracy
Figures within 1% of XBRL, across a golden set of 10-Ks
100.0%
One run so far, Aug 14, 2026
Q&A
Retrieval-augmented answers over a labelled question set
The question set grew between runs, so each rate is over that run's own questions.
Runs
| Date | Model | Filings | Accuracy |
|---|---|---|---|
| Aug 14, 2026 | gemini-3.6-flash | 10 | 100.0% (40/40) |
| Date | Model | K | Questions | Grounded | Refused | Hit @K | Citation |
|---|---|---|---|---|---|---|---|
| Oct 10, 2026 | gemini-3.5-flash-lite | 6 | 41 | 100.0% | 100.0% | 91.4% | 43.8% |
| Oct 8, 2026 | gemini-3.5-flash-lite | 6 | 26 | 100.0% | 100.0% | 85.0% | 47.5% |
| Sep 14, 2026 | gemini-3.5-flash-lite | 6 | 20 | 100.0% | 100.0% | 86.7% | 37.8% |
Method
A grounded answer states no figure that is missing from its excerpts, other than one a single arithmetic step derives from two that are present. A refusal counts only when the model declines and invents nothing while declining. Hit rate asks whether a hand-labelled passage was among the K excerpts retrieved. Citation precision is the share of those excerpts the answer drew on. The build fails when grounding, refusal or extraction accuracy falls below its committed floor.