Benchmarks

SimpleQA: who claims what

Everyone quotes a SimpleQA score. Few publish the run. We opened every vendor's benchmark pages and GitHub in a browser and logged each claim, who measured it, and whether you can check it. A score you can't reproduce is marketing until proven otherwise.

• No published score found: Jina, Mojeek, Bocha, every SERP scraper except Serper.
• Who runs the test matters: Brave scores 94.1 F1 in its own run, 87% in Parallel's and 76% in Tavily's. Tavily reports 93.3% for itself and gets 88.5% in You.com's. Trust the public repos, and trust your own bake-off most.
• Relative-only claims: TinyFish publishes "3.6 pts ahead on SimpleQA" with no absolute score. Parallel's 2025 launch post reported 98% for an earlier release.
• Not apples to apples: some vendors report accuracy, others an F-score, and some use the 1,000-question SimpleQA Verified subset. The model that reads the results and the grader prompt also differ from run to run.
• Keirolabs measures presence, not answers: its deep (200 questions) and lite (500 questions) repos score whether the answer appears in what comes back, with an LLM judge for variant forms. They don't have a model answer the question. Lite runs vary by about ±3 points between runs, and the 92.0% recipe reads about 37,000 tokens per question.
• Contamination: SimpleQA answers now sit on pages all over the web, so a search API can find the benchmark itself. High scores partly measure that.