Web Search API Benchmark 2026: Tavily vs Exa vs Parallel vs Brave vs SerpApi vs Perplexity vs Firecrawl vs You.com vs Keenable vs cloro
Perplexity, cloro, Parallel, Keenable and Exa are the most accurate web search APIs for AI agents on short factual questions in this benchmark of 2,182 questions, and the order changes with the question type: cloro scores highest on recent events, and cloro and SerpApi lead on multi-hop questions by 27 to 64 points.
Accuracy by question type, one search per question, with the 95% interval in brackets:
| Search API | Short factual: SimpleQA Verified (1,000) | Recent events: FreshQA (358) | Multi-hop: FRAMES (824) |
|---|---|---|---|
| Perplexity Search API* | 96.9% (95.6-97.8) | 86.0% (82.1-89.2) | 44.2% (40.8-47.6) |
| cloro | 96.4% (95.1-97.4) | 88.5% (84.8-91.4) | 79.9% (77.0-82.5) |
| Parallel | 96.2% (94.8-97.2) | 86.0% (82.1-89.2) | 45.6% (42.3-49.0) |
| Keenable‡ | 96.0% (94.6-97.0) | 84.4% (80.2-87.8) | 34.5% (31.3-37.8) |
| Exa | 95.8% (94.4-96.9) | 86.6% (82.7-89.7) | 49.9% (46.5-53.3) |
| SerpApi | 93.9% (92.2-95.2) | 86.9% (83.0-90.0) | 76.8% (73.8-79.6) |
| Brave Search API | 90.5% (88.5-92.2) | 85.8% (81.8-89.0) | 25.8% (23.0-28.9) |
| You.com† | 89.2% (87.1-91.0) | 84.9% (80.8-88.3) | 22.1% (19.4-25.0) |
| Tavily | 89.2% (87.1-91.0) | 82.7% (78.4-86.2) | 18.1% (15.6-20.9) |
| Firecrawl | 88.8% (86.7-90.6) | 80.2% (75.7-84.0) | 15.8% (13.4-18.4) |
* Perplexity's searches exclude huggingface.co and kaggle.com, which host copies of the benchmark datasets. In its first run, without the filter, its results held the answer from a dataset page on 471 of 1,000 SimpleQA Verified questions. The other APIs return such a page on a smaller share of questions and run without the filter; each report has the leak check. The first Perplexity run is in its report, and the filtered run in this report.
† You.com rejects queries longer than 50 words (HTTP 422): 42 FRAMES questions and 1 SimpleQA Verified question. They count as not attempted. Without them You.com scores 23.3% on FRAMES. See the You.com report.
‡ Keenable's searches also exclude huggingface.co and kaggle.com. Without the filter, its results held a dataset page on 206 of 824 FRAMES questions, and with the earlier reader it scored 88.3% on those and 59.1% on the others. See the Keenable report.
All scores come from the reader gemini-3.8-flash with a prompt that forbids
the reader's own knowledge. How the reader was chosen
says why, and the rerun report has the change
from the earlier reader.
An open benchmark of ten web search APIs for AI agents and RAG pipelines:
accuracy, cost per 1,000 searches and latency of cloro, SerpApi, Tavily, Exa,
Parallel, Brave Search API, Perplexity Search API, Firecrawl, You.com and
Keenable, on SimpleQA
Verified (1,000 questions), FreshQA (358 questions) and FRAMES (824 multi-hop
questions). Every provider got one search per question, the same reader,
gemini-3.8-flash, and the same grader, gemini-pro-latest, in November 2026. The code,
the settings and every graded answer are in this repo.
- Most accurate on SimpleQA Verified: Perplexity 96.9%, cloro 96.4%, Parallel 96.2%, Keenable 96.0% and Exa 95.8%. The five are within the margin of error of each other. SerpApi scored 93.9%, Brave 90.5%, You.com and Tavily 89.2%, and Firecrawl 88.8%.
- Most accurate on FreshQA (questions whose answer changes over time): cloro 88.5%, then SerpApi 86.9%, Exa 86.6%, Parallel and Perplexity 86.0%, Brave 85.8%, You.com 84.9%, Keenable 84.4%, Tavily 82.7% and Firecrawl 80.2%. The gaps from cloro to the next five are not significant.
- Most accurate on FRAMES (multi-hop questions that need several sources): cloro 79.9%, then SerpApi 76.8%, Exa 49.9%, Parallel 45.6%, Perplexity 44.2%, Keenable 34.5%, Brave 25.8%, You.com 22.1%, Tavily 18.1% and Firecrawl 15.8%.
- Lowest cost per 1,000 searches: Keenable's free tier covers 100,000 searches a month, so at that volume it costs nothing. Past the free tiers, cloro costs $1.92 on a plan for 100,000 searches a month, then Firecrawl $3.49, Keenable $4, Parallel, Brave, Tavily, Perplexity and You.com at $5, Exa at $7 and SerpApi at $7.25. SerpApi needed two searches on most questions.
- Fastest (p50 search latency): Keenable 0.27 s, Perplexity 0.34 s, Exa 0.74 s, You.com 0.83 s, Firecrawl 0.93 s, Brave 1.21 s, Tavily 2.00 s, Parallel 2.99 s. cloro (7.96 s) and SerpApi (8.20 s) are the slowest.
On SimpleQA Verified, 1,000 short factual questions, the most accurate web search APIs in this benchmark were Perplexity (96.9%), cloro (96.4%), Parallel (96.2%), Keenable (96.0%) and Exa (95.8%). A paired test on the same questions finds no significant difference between any two of them (p = 0.071 to 0.88). Each of the five was significantly more accurate than SerpApi (93.9%, p = 0.00023 to 0.037), Brave Search API (90.5%), You.com (89.2%), Tavily (89.2%) and Firecrawl (88.8%), each at p < 0.0001. Each API got one search per question, and the same model answered from its results.
Which API is "most accurate" depends on the question type. On questions about recent events (FreshQA), cloro scores highest, and five APIs are within the margin of error of it. On multi-hop questions (FRAMES), cloro (79.9%) and SerpApi (76.8%) lead, and Exa, Parallel, Perplexity, Keenable, Brave, You.com, Tavily and Firecrawl answer "not found" on 372 to 673 of 824 questions.
Accuracy with the 95% Wilson interval, one search per question. Cost is the list price per 1,000 searches on the plan that covers 100,000 searches a month. Latency is the p50 of the search request alone, on SimpleQA Verified.
| Search API | SimpleQA Verified (1,000) | FreshQA (358) | FRAMES (824) | $ per 1,000 searches | p50 latency |
|---|---|---|---|---|---|
| Perplexity Search API* | 96.9% (95.6-97.8) | 86.0% (82.1-89.2) | 44.2% (40.8-47.6) | $5.00 | 0.34 s |
| cloro | 96.4% (95.1-97.4) | 88.5% (84.8-91.4) | 79.9% (77.0-82.5) | $1.92 | 7.96 s |
| Parallel | 96.2% (94.8-97.2) | 86.0% (82.1-89.2) | 45.6% (42.3-49.0) | $5.00 | 2.99 s |
| Keenable‡ | 96.0% (94.6-97.0) | 84.4% (80.2-87.8) | 34.5% (31.3-37.8) | $4.00 | 0.27 s |
| Exa | 95.8% (94.4-96.9) | 86.6% (82.7-89.7) | 49.9% (46.5-53.3) | $7.00 | 0.74 s |
| SerpApi | 93.9% (92.2-95.2) | 86.9% (83.0-90.0) | 76.8% (73.8-79.6) | $7.25 | 8.20 s |
| Brave Search API | 90.5% (88.5-92.2) | 85.8% (81.8-89.0) | 25.8% (23.0-28.9) | $5.00 | 1.21 s |
| You.com† | 89.2% (87.1-91.0) | 84.9% (80.8-88.3) | 22.1% (19.4-25.0) | $5.00 | 0.83 s |
| Tavily | 89.2% (87.1-91.0) | 82.7% (78.4-86.2) | 18.1% (15.6-20.9) | $5.00 | 2.00 s |
| Firecrawl | 88.8% (86.7-90.6) | 80.2% (75.7-84.0) | 15.8% (13.4-18.4) | $3.49 | 0.93 s |
* and ‡ With huggingface.co and kaggle.com excluded. † Queries over 50 words count as not attempted. See the notes under the first table.
The scores, the change from the earlier reader and the paired tests are in
the rerun report. The earlier reports, with
the leak check, the questions with possibly wrong gold answers, the run costs
and the scores of the earlier reader, are in runs/:
SimpleQA Verified,
FreshQA,
FRAMES, and You.com and Keenable in their own
reports (You.com,
Keenable). The Gemini grades and the Firecrawl
runs on SimpleQA Verified and FreshQA are in the
regrade report. The tests of which parts of a
search result give the accuracy are in the
sensitivity report.
On FreshQA, 358 questions whose answers can change over time (a third of them with a false premise), cloro (88.5%) scored highest. The gaps to SerpApi (86.9%, p = 0.29), Exa (86.6%, p = 0.28), Parallel (86.0%, p = 0.11), Perplexity (86.0%, p = 0.14) and Brave Search API (85.8%, p = 0.087) are not significant. cloro was significantly more accurate than You.com (84.9%, p = 0.024), Keenable (84.4%, p = 0.020), Tavily (82.7%, p = 0.0008) and Firecrawl (80.2%, p < 0.0001).
On FRAMES, 824 questions that need facts from two or more Wikipedia pages, cloro (79.9%) and SerpApi (76.8%) were the most accurate web search APIs, and cloro was ahead of SerpApi in the paired test (68 wins to 43, p = 0.022). Both were ahead of Exa (49.9%), Parallel (45.6%), Perplexity (44.2%), Keenable (34.5%), Brave Search API (25.8%), You.com (22.1%), Tavily (18.1%) and Firecrawl (15.8%), each at p < 0.0001. With one search, those eight answered "not found" on 372 to 673 of the 824 questions, against 28 for cloro and 62 for SerpApi. When they answered, the eight were correct 86.1% to 91.0% of the time, and cloro and SerpApi 82.7% and 83.1%. So the gap comes from how many questions one results page can answer at all. The results of cloro and SerpApi usually include Google's AI Overview, which often holds the full answer. FRAMES is usually run with an agent that searches many times; this run measures one search.
Up to 100,000 searches a month, the cheapest web search API in this benchmark is Keenable: its free tier covers that volume, so it costs nothing. Past the free tiers, the cheapest is cloro, at $1.92 per 1,000 searches on a plan for 100,000 searches a month, and Keenable costs $4. At 200,000 searches a month Keenable costs $400, $2.00 per 1,000 on average. The table gives each provider's cheapest self-serve plan that covers 100,000 searches a month, the plan price divided by what the plan includes (list prices read on November 5, 2026):
| Search API | $ per 1,000 searches | Plan and request | Free tier |
|---|---|---|---|
| cloro | $1.92 | Starter, $250 a month for 650,000 credits, 5 credits a search (async) | 500 credits a month (100 searches) |
| Parallel | $5.00 | Search API, pay as you go, default mode | 5,000 requests a month |
| Brave Search API | $5.00 | Search, $5 per 1,000 requests | $5 of credits a month (1,000 requests) |
| Tavily | $5.00 | Growth, $500 a month for 100,000 basic searches | 1,000 credits a month |
| Exa | $7.00 | Search (auto), pay as you go | $10 of credits a month |
| SerpApi | $7.25 | Searcher, $725 a month for 100,000 searches | 250 searches a month |
| Perplexity Search API | $5.00 | $5 per 1,000 requests (read November 18) | Not found |
| Firecrawl | $3.49 | Standard, $99 a month for 100,000 credits plus 100,000 extra credits at $5 per 2,000; 2 credits a search (read November 18) | 1,000 credits a month (500 searches) |
| You.com | $5.00 | Web Search API, $5 per 1,000 calls, pay as you go (read November 18) | $10 of credit (about 2,000 searches) |
| Keenable | $4.00 | Pay as you go, $4 per 1,000 requests past the free allowance (read November 20) | 100,000 requests a month |
SerpApi bills a second search when part of the result loads separately. That
happened on 97% of SimpleQA questions and 75% of FreshQA questions, so a
SimpleQA question cost about 1.97 searches, $14.28 per 1,000 questions. A sync
cloro request costs 7 credits instead of 5, $2.69 per 1,000 searches. Exa's
price matches the costDollars field of its own responses
($0.007 a request with highlights). Prices change: check each provider's
pricing page before you buy.
Keenable has the largest free tier in this benchmark: 100,000 requests a month, with no card. Of the six web search APIs of the first run, which all have a free tier, Exa's monthly tier buys the most searches with the request used here: $10 of credits a month, about 1,428 searches at $7 per 1,000. Of the five that say whether they need a card, Parallel and Brave do.
| Search API | Free allowance | Refill | Searches with the benchmark's request | Card needed |
|---|---|---|---|---|
| Keenable | 100,000 requests (read November 20) | Monthly, on the day the organization was created | 100,000 | No (a signup from a disposable or privacy mail domain gets 1,000 until a card is saved) |
| Exa | $10 of credits, plus $10 once after onboarding | Resets to $10 on the 1st of each month | About 1,428 (auto, $7 per 1,000) |
No |
| Parallel | $5 of credits with a card on file, up to 5,000 requests; up to $80 once at sign-up | Monthly ($5) | 1,000 in the default mode ($5 per 1,000) | Yes, for the $5 |
| Brave Search API | $5 of credits on the Search plan | Monthly | 1,000 ($5 per 1,000) | Yes, "to prevent abuse" |
| Tavily | 1,000 API credits | Monthly | 1,000 basic searches (1 credit each) | No |
| SerpApi | 250 searches, 50 an hour | Monthly | 250 (a SimpleQA question took about 1.97 searches) | Not stated |
| Firecrawl | 1,000 credits (read November 18) | Monthly | 500 (2 credits a search) | No |
| You.com | $10 of credit (read November 18) | Not stated | About 2,000 ($5 per 1,000) | Not stated |
| cloro | 500 credits, 1 request at a time | Monthly | 100 at 5 credits (Google Search with the AI Overview); 166 Google searches without it, at 3 credits each | No |
Sources, all read on November 16, 2026: exa.ai/pricing, parallel.ai/pricing, brave.com/search/api, tavily.com/pricing and docs.tavily.com/documentation/api-credits, serpapi.com/pricing, cloro.dev/pricing and the cloro billing guide. A sync cloro request adds 2 credits, so 500 credits make 71 sync searches with the AI Overview. The Perplexity pricing page shows no free tier for the Search API (read November 18, 2026). you.com/pricing gives "$10 free credit = 2,000 searches" and does not say if the credit renews (read November 18, 2026). keenable.ai/pricing and docs.keenable.ai/credits.md give the Keenable allowance (read November 20, 2026).
A free tier answers "What is the cheapest web search API?" only up to these volumes. Past them, the number to compare is the price per 1,000 searches in How much does a web search API cost per 1,000 searches?.
cloro had the lowest cost per correct answer on SimpleQA Verified and FreshQA. cloro costs $1.99 per 1,000 correct SimpleQA answers ($1.92 / 96.4%), against $3.93 for Firecrawl, $4.17 for Keenable, $5.16 for Perplexity, $5.20 for Parallel, $5.52 for Brave, $5.61 for You.com and Tavily, $7.31 for Exa and $15.21 for SerpApi (1.97 searches a question). On FreshQA, cloro and SerpApi are not significantly different, and cloro costs 73% less per search. These figures use the price past the free tier. Inside Keenable's free tier, 100,000 searches a month, a correct answer from Keenable costs nothing.
Keenable was the fastest web search API in this benchmark, with a p50 search latency of 0.27 s on SimpleQA Verified. Perplexity Search API took 0.34 s, Exa 0.74 s, You.com 0.83 s, Firecrawl 0.93 s, Brave Search API 1.21 s, Tavily 2.00 s and Parallel 2.99 s. cloro (7.96 s) and SerpApi (8.20 s) were the slowest. These figures come from runs with no other benchmark at the same time, from one machine, 4 requests in parallel. Firecrawl ran 1 request at a time, because its rate limit returned HTTP 429 at 4. The Keenable figure comes from a search-only run with no other benchmark at the same time. Latency is the search request only, without the LLM. If you need answers in under 3 seconds, cloro and SerpApi are not the right choice today.
Paired exact McNemar tests on the same questions. "Wins" is the number of questions one API answered correctly and the other did not.
| Comparison | SimpleQA Verified | FreshQA | FRAMES |
|---|---|---|---|
| Exa vs Tavily | Exa, 81 wins to 15 (p < 0.0001) | Exa, 28 to 14 (p = 0.044) | Exa, 285 to 23 (p < 0.0001) |
| Brave vs Tavily | no difference (p = 0.29) | no difference (p = 0.099) | Brave, 125 to 61 (p < 0.0001) |
| Exa vs Parallel | no difference (p = 0.63) | no difference (p = 0.81) | Exa, 120 to 85 (p = 0.017) |
| Exa vs Brave | Exa, 72 to 19 (p < 0.0001) | no difference (p = 0.66) | Exa, 218 to 20 (p < 0.0001) |
| Parallel vs Tavily | Parallel, 87 to 17 (p < 0.0001) | no difference (p = 0.058) | Parallel, 265 to 38 (p < 0.0001) |
| Firecrawl vs Tavily | no difference (p = 0.78) | no difference (p = 0.21) | no difference (p = 0.13) |
| cloro vs SerpApi | cloro, 38 to 13 (p = 0.0006) | no difference (p = 0.29) | cloro, 68 to 43 (p = 0.022) |
| cloro vs Tavily | cloro, 88 to 16 (p < 0.0001) | cloro, 29 to 8 (p = 0.0008) | cloro, 521 to 12 (p < 0.0001) |
| cloro vs Brave | cloro, 77 to 18 (p < 0.0001) | no difference (p = 0.087) | cloro, 457 to 12 (p < 0.0001) |
| cloro vs Exa | no difference (p = 0.46) | no difference (p = 0.28) | cloro, 280 to 33 (p < 0.0001) |
| cloro vs Parallel | no difference (p = 0.88) | no difference (p = 0.11) | cloro, 310 to 28 (p < 0.0001) |
| cloro vs Perplexity | no difference (p = 0.51) | no difference (p = 0.14) | cloro, 333 to 39 (p < 0.0001) |
| cloro vs Firecrawl | cloro, 93 to 17 (p < 0.0001) | cloro, 40 to 10 (p < 0.0001) | cloro, 535 to 7 (p < 0.0001) |
| cloro vs You.com | cloro, 88 to 16 (p < 0.0001) | cloro, 21 to 8 (p = 0.024) | cloro, 483 to 7 (p < 0.0001) |
| You.com vs Brave | no difference (p = 0.060) | no difference (p = 0.58) | Brave, 45 to 14 (p < 0.0001) |
| You.com vs Tavily | no difference (p = 1.0) | no difference (p = 0.22) | You.com, 99 to 66 (p = 0.012) |
| cloro vs Keenable | no difference (p = 0.64) | cloro, 26 to 11 (p = 0.020) | cloro, 404 to 30 (p < 0.0001) |
| Keenable vs Exa | no difference (p = 0.86) | no difference (p = 0.15) | Exa, 191 to 64 (p < 0.0001) |
| Keenable vs Perplexity | no difference (p = 0.15) | no difference (p = 0.29) | Perplexity, 156 to 76 (p < 0.0001) |
| Keenable vs Brave | Keenable, 74 to 19 (p < 0.0001) | no difference (p = 0.46) | Keenable, 136 to 65 (p < 0.0001) |
Exa was more accurate than Tavily on all three datasets: SimpleQA Verified (95.8% against 89.2%), FreshQA (86.6% against 82.7%, p = 0.044) and FRAMES (49.9% against 18.1%). Exa was also faster (0.74 s p50 against 2.00 s) and cost more ($7 against $5 per 1,000 searches).
In this benchmark cloro beat SerpApi on SimpleQA Verified and FRAMES and matched it on FreshQA, at about a quarter of the price per search: 96.4% against 93.9% on SimpleQA Verified (paired p = 0.0006), 88.5% against 86.9% on FreshQA (no significant difference, p = 0.29) and 79.9% against 76.8% on FRAMES (p = 0.022), at $1.92 against $7.25 per 1,000 searches, and SerpApi needed a second search on most questions. Both are slow: cloro's p50 latency was 7.96 s and SerpApi's 8.20 s.
This benchmark is a small RAG pipeline: one search, the top 10 results into the context window, one LLM answer grounded only on those results. For factual questions, Perplexity, cloro, Parallel, Keenable and Exa gave the most correct answers. For time-sensitive questions, cloro scored highest, with SerpApi, Exa, Parallel, Perplexity and Brave within the margin of error. For multi-hop questions, cloro and SerpApi did. For the lowest cost per correct answer, cloro. For latency under 1 second, Keenable, Perplexity, Exa or You.com. For the most free searches, Keenable (100,000 a month). Run the harness on 50 to 100 of your own questions before you choose, because the order changed between the three datasets.
We kept cloro's saved responses for 100 SimpleQA Verified questions and
changed only what the reader could read. The AI Overview gave most of the
accuracy: 99% with it, 86% without it (13 questions lost, none gained,
p < 0.001). The snippets gave most of the rest: the AI Overview and the
snippets alone scored 99% with 37% fewer reader tokens. Titles, links, results
6 to 10, a second results page, the full JSON response and the text of the
result pages added tokens and no significant accuracy. A second reader model and a second
grader gave the same results. These tests ran with the earlier reader,
gpt-5.6-luna, before the strict prompt. The tables, the preliminary FreshQA
numbers and the commands are in the
sensitivity report.
Every provider got the same setup:
- One search per question. The query is the question text, unchanged. There is no query rewrite, no second search and no page fetch.
- 10 results, in the provider's default search mode.
- The same reader.
gemini-3.8-flashreads at most 30,000 characters of search results and gives a short answer from those results only, or "NOT FOUND". The prompt says: "Use ONLY information from the search results. Do NOT use any internal knowledge." For FreshQA and FRAMES the prompt gives today's date, and for FreshQA it asks the reader to point out a false premise. - The same grader. Gemini (
gemini-pro-latest) grades each answer CORRECT, INCORRECT or NOT_ATTEMPTED with the simple-evals grader prompt (grader_template.txt), not changed. - The same questions, shuffled with seed 42.
| Search API | Request | Text given to the reader |
|---|---|---|
| cloro | Search endpoint, country US, 5 credits | Search results with titles, links, snippets and answer blocks |
| SerpApi | engine=google, gl=us, hl=en, no_cache=true, plus the follow-up request when part of the page loads separately |
Answer blocks, knowledge graph, organic results, related questions |
| Tavily | search_depth=basic |
title, url, content |
| Exa | type=auto, highlights |
title, url, highlights |
| Parallel | default mode, objective and query = the question | title, url, excerpts |
| Brave Search API | extra_snippets=true |
title, url, description, extra snippets |
| Perplexity Search API | /search, max_results=10, huggingface.co and kaggle.com excluded |
title, url, snippet |
| Firecrawl | /v2/search, limit=10, no page scraping |
title, url, description |
| You.com | /v1/search, count=10 |
title, url, description, snippets |
| Keenable | /v1/search, max_results=10, default mode (pro), huggingface.co and kaggle.com excluded |
title, url, snippet |
The answer features that providers sell as separate options (Tavily
include_answer, Exa summary) are off, because they are not part of a
default search response.
Dropped rows: FreshQA uses the April 21, 2026 test split without fast-changing rows and rows past their review date, which leaves 358.
Metrics: accuracy is correct answers over all questions, with a failed search
counted as not attempted. Latency is the time of the search request from the
first byte sent to the last byte received, the sum of both requests when a
provider needs two. All providers ran from the same machine. The first six ran
in the same time window, interleaved question by question, 4 requests at a
time. Perplexity, Firecrawl, You.com and Keenable ran later in November 2026.
On November 21, 2026, gemini-3.8-flash read the saved responses of all ten
APIs again, with no new searches. The tables come from that run.
The reader must answer from the search results only. An answer from its own memory is not a result of the search API. So the prompt has the line "Use ONLY information from the search results. Do NOT use any internal knowledge.", and the reader is the model that keeps to that line best.
We gave 11 reader models the same saved search results for 400 FRAMES
questions. Two judges, gpt-5.6-sol and gemini-pro-latest, checked each
answer: do the search results alone support it?
| Reader | Answers not supported by the search results |
|---|---|
gpt-4.1 |
27% |
gpt-5.6-luna (the earlier reader) |
23% |
gemini-3.1-flash-lite |
18% |
gemini-3.8-flash |
4% |
| Seven other Gemini models | 2% to 4% |
The GPT readers gave the most correct answers on FRAMES because they added
facts from memory. gemini-3.8-flash is the least expensive reader that
keeps to the results and uses them fully. The change lowered the FRAMES
scores of cloro and SerpApi by less than 2 points and those of the other
eight APIs by 14 to 30 points. cloro runs this benchmark and gains from the
change, so the reader test has every
answer and both judges' verdicts.
- cloro runs this benchmark and sells one of the tested APIs. That is why the code, the prompts, the settings and the graded answers are public. If a setting is not fair to a provider, change it and run it again.
- The reader can answer from its own memory. The prompt forbids it, and
gemini-3.8-flashkeeps to that: 4% of its FRAMES answers in the reader test were not supported by the search results. A reader that adds facts from memory does not affect every API the same way. It lifts the APIs whose results hold part of the answer. - Some pages copy benchmark answers. Every API returned a page of a benchmark dataset on some questions. On SimpleQA Verified the first six APIs did so on 23 to 117 questions each, Firecrawl on 81, You.com on 120, Keenable, without its filter, on 101 and Perplexity, without its filter, on 500. On FRAMES, Keenable did so on 206 of 824 questions without its filter, and on 46 with it, through Hugging Face mirrors. With the earlier reader, the order of the first six did not change without the flagged questions. Each report has the leak check.
- Gold answers go stale. On 30 FreshQA questions most APIs disagreed with an answer key that may be out of date (earlier reader). Without them the order did not change.
- One search per question measures the search API, not an agent. A multi-step agent that searches many times can get a high score from a weak search API.
Python 3.10 or later, no packages to install. Set GEMINI_API_KEY and the
keys of the providers you want to test:
| Provider | Key |
|---|---|
| cloro | CLORO_API_KEY |
| SerpApi | SERPAPI_KEY |
| Tavily | TAVILY_API_KEY |
| Exa | EXA_API_KEY |
| Parallel | PARALLEL_API_KEY |
| Brave | BRAVE_API_KEY |
| Perplexity | PERPLEXITY_API_KEY |
| Firecrawl | FIRECRAWL_API_KEY |
| You.com | YDC_API_KEY |
| Keenable | KEENABLE_API_KEY |
The reader defaults to gemini-3.8-flash and the grader to
gemini-pro-latest, the models of the published tables. A run with another
reader is not comparable with the tables: on FRAMES, the reader changed the
scores of eight APIs by 14 to 30 points. Every published grade is a Gemini
grade: do not grade with another model.
python3 benchmark.py --providers cloro,serpapi,tavily,exa,parallel,brave --n 1000 --workers 4Run --n 3 first to find a broken key before you spend the budget.
--dataset simpleqa|verticalrtk|freshqa|frames|sealqa|finsearchcomp|obnewsselects the dataset (defaultsimpleqa).--as-of YYYY-MM-DDsets the date in the reader prompt and the FreshQA filter (default: today). Use the date of an earlier run to read its saved responses again with the same questions.- The cloro variants
cloro-reread,cloro-drop-aio,cloro-organic,cloro-json,cloro-top5,cloro-aio-only,cloro-aio-last,cloro-no-snippet,cloro-no-url,cloro-no-title,cloro-minimalandcloro-2p-page1make no search call. They read the saved raw responses ofcloro(orcloro-2p) in the same--outfolder, so run that provider first.cloro-2pasks for two Google result pages. <provider>-reread(for every API except cloro, which hascloro-reread) reads the saved raw responses of<provider>again in the published format, with no search call. Use it to test another reader model on the same results, as in the reader test.cloro-othergives each question the saved cloro results of a different question. A reader that keeps to its input replies NOT FOUND. It tests the reader, not an API.--search-onlymeasures latency with no LLM calls.--out <folder>sets the run folder. Results go to<folder>/results/<provider>.jsonland raw responses to<folder>/raw/<provider>/<question id>.json.- A run that stops continues when you run the same command again. Rows with a search or LLM error run again. A 402 from a provider does not stop the run: the harness writes an ERROR row and continues, so watch the log.
python3 benchmark.py --summary --out runs/simpleqa-2026-11prints the results table of a run again.python3 analysis/analyze.py <run folder> <dataset>prints per-topic accuracy, the questions that 4 or more providers miss, and the leak check (all datasets exceptsimpleqa).python3 analysis/sensitivity.py <run folder> cloro:cloro-organiccompares two result files on the same questions, with an exact McNemar test.
A full SimpleQA run (1,000 questions, 6 providers) used about 7,000 cloro credits, 1,970 SerpApi searches, 1,000 Tavily credits, $7.00 of Exa, 1,000 Parallel and 1,000 Brave requests, and 12,000 LLM calls.
To add a provider, add a search function to benchmark.py
and add it to PROVIDERS. The function returns the raw response and a list
of text blocks for the reader.
This one is run by cloro, a vendor, and it is reproducible: the harness,
the questions, the settings and all 21,820 graded answers are in this repo.
Each provider got the same single search, the same reader (gemini-3.8-flash)
and the same grader, so you can check every result and run it again with your
own keys.
One search per question measures the search API, not the agent. Many published scores let an agent search many times and open pages, which hides the difference between search APIs.
SimpleQA Verified (1,000 short factual questions), FreshQA (358 questions whose answers change over time, including false-premise questions) and FRAMES (824 multi-hop questions that need facts from several Wikipedia pages).
One that answers only from the search results. With a prompt that forbids
outside knowledge, gpt-5.6-luna and gpt-4.1 still gave 23% and 27% of
their FRAMES answers from memory, and gemini-3.8-flash 4%. A reader that
fills gaps from memory hides weak search results and shrinks the gaps
between APIs. See How the reader was chosen.
The AI Overview. On 100 SimpleQA Verified questions, the reader answered 99% correctly with it and 86% without it, from the same results. The snippets carry most of the rest. See the sensitivity report.
In November 2026. The first six APIs ran SimpleQA Verified on November 4 and
5, FreshQA on November 5 and FRAMES on November 10 and 11. Perplexity and
Firecrawl ran on November 18, and Gemini graded every answer on the same day.
You.com ran on November 18 and 19, and Keenable on November 20, both graded
by Gemini. On November 21, gemini-3.8-flash read the saved responses of
all ten APIs again, and the tables come from that run. Prices were read on
November 5, on November 18 for Perplexity, Firecrawl and You.com, and on
November 20 for Keenable.
cloro, "Web Search API Benchmark 2026", November 2026, https://github.com/cloro-dev/web-search-api-benchmark. The results and the sensitivity tests are written up at cloro.dev/blog/web-search-api-benchmark, and compared with other search APIs at cloro.dev/blog/best-search-api.
The code and the reports are under the MIT License. The files in
runs/*/results/ hold the question and answer text of three public datasets.
That text stays under the license of its source:
| Dataset | Source | License | Changes in this repo |
|---|---|---|---|
| SimpleQA Verified | google/simpleqa-verified | MIT | None |
| FreshQA | freshllms/freshqa, version of April 21, 2026 | Apache-2.0 (repository) | Test split only, without the fast-changing questions and the questions past their review date |
| FRAMES | google/frames-benchmark, test.tsv |
Apache-2.0 | None |
If you use these datasets, cite their authors as their pages ask: the SimpleQA Verified technical report, the FreshLLMs paper (arXiv:2310.03214) and the FRAMES paper (arXiv:2409.12941).