Benchmark report

Task verified

4m 5s elapsed

The saved attempt met the documented success criterion.

Experience score

90/ 100

Agent-Ready
  • SInstall
  • SAuth
  • AExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute A, Docs S, Current SSInstallSAuthAExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Tavily's crawl response exposes no title field, so titles had to be mined from raw_content markdown headings, costing one extra run.

Where the time went

4:05 agent time
  • Research0:23
  • Attempt2:07
  • Verify1:35
4m 5sTotal elapsed
1:20Time to first success
0:00Waited on the owner

Judge reason

Live Tavily Crawl verified from saved artifacts. tavily-crawl/request.json shows url=https://www.postgresql.org/docs/17/tutorial.html, limit=25, and the documented include option select_paths=["/docs/17/tutorial.*"] plus select_domains/allow_external=false, so the restriction is server-side. crawl.py does one POST https://api.tavily.com/crawl with Bearer auth (key read from the benchmark-provided file), no per-page fetching or local link-following, and no local URL filtering (only exact-URL de-dup). response_meta.json records HTTP 200 with live CloudFront headers dated Mon, 16 Nov 2026 15:49:25 GMT; response.json carries request_id 71e786e3-7e4c-4f10-9b79-a8a50b1526c6, response_time 0.72s, usage 7 credits, and 23 results. I re-parsed response.json: 23 distinct URLs, zero outside the https://www.postgresql.org/docs/17/tutorial prefix, and pages.json/run_output.txt URL set matches the response exactly. Titles are not fabricated: re-running the heading-extraction logic over raw_content reproduces every title in pages.json byte-for-byte, and each page's raw_content independently corroborates it via the site's own "PostgreSQL 17 - <title>" link text (e.g. tutorial-arch.html -> "1.2. Architectural Fundamentals"). raw_content is genuine live content (averages ~5.6 KB, includes the Oct 15 2026 PostgreSQL 19 RC 1 banner). 23 pages is within the 10-25 band and under the cap. Docs were correctly identified as generally available (no beta/preview marking), and the endpoint is synchronous so no polling applied.

Attempt record

  1. 0:52Answered from saved key: Tavily API keyno wait
  2. —First success verified

Attempt artifacts saved.