Gemini agent vs production: 40-question benchmark

Rettsdata AI · run 23 September 2026 · production = main @ 01b8224 · candidate = staging @ f0e07c1

What was compared

Both systems ran their real code paths from the respective commits, headless (no database writes), same machine, same concurrency (3 at a time each, interleaved), same 40 questions.

Answer quality

Blind pairwise preference

Each judge saw both answers as A and B in a randomised order, with citations renumbered to the same [k] style so the format didn't reveal the system. The two judges picked the same winner on 90% of questions.

Judge self-preference check: GPT-5.5 preferred the candidate 37 of 40 times and Gemini 3.1 Pro 39 of 40, so the Gemini judge was not meaningfully more favourable to the Gemini answers.

Accuracy on 15 gold questions

Settled questions with a fixed answer key (deadline, amount or age plus the governing provision). Score = share of key facts stated correctly (partial = half), averaged over both judges. ⚑ = number of judges that flagged at least one incorrect legal statement elsewhere in the answer.

Incorrect-statement rate (answers flagged by a judge): production 40%, candidate 13%. Production's flags were mostly wrong sub-sections (e.g. the wrong ledd of aml. § 15-3) and invented exceptions (e.g. a one-week appeal deadline in tvl. § 29-5); the candidate's were imprecise side remarks.

Grounding and hallucination

Each judge listed up to 15 material legal claims per answer and checked each against the text of the source it cites (not against its own knowledge). “Unsupported” means the cited source does not say it, which is the closest measurable proxy for hallucination. 854 claims graded for production, 879 for the candidate.

Speed

Wall-clock from sending the question. “First text” is the first visible answer token. The candidate shows its search steps in the UI before that, production shows status text.

The candidate is slower mainly because it reads more: 66 retrieved chunks and 73k input tokens per answer vs 20 chunks and 20k, plus ~920 thinking tokens, and it writes longer answers (median 2545 vs 1721 characters).

Behaviour and cost

Prices: GPT-4.1 $2 / $8 per 1M tokens plus ~$0.002 per Cohere rerank; Gemini 3.7 Flash $0.75 / $3.75 (introductory, doubling on 1 Jan 2027), Priority PayGo taken as +87.5% (Google quotes 75–100%). Production's cost covers only the answer call; its query-generation and evaluation calls were not counted, so production is somewhat understated.

Method and caveats

• Questions: 15 gold questions with a written answer key, and 25 real questions sampled from production traffic (not shown here because they are customer data). Standalone first messages only; no conversation history.
• Judges: GPT-5.5 (Azure) and Gemini 3.1 Pro (Vertex), temperature 0, results averaged. Using two vendors limits self-preference; the Gemini judge was not noticeably kinder to the Gemini system.
• Single run per question. Both systems are non-deterministic; with 40 questions small differences (e.g. ±5 points on a percentage) are within noise. The quality gap is far larger than that; the latency gap is consistent across the p50 and p90.
• The answer key was written for this benchmark and covers settled rules, so it measures basic correctness, not hard legal reasoning. The real-question comparison covers the harder end.
• Grounding is judged against cited chunk text only, truncated to 2,500 characters per source. A claim that is true but cited to the wrong chunk counts as unsupported.
• The expand tool was never called on these 40 questions, so it is not what drives the difference.