Benchmark report

Task verified

6m 28s elapsed

The saved attempt met the documented success criterion.

Experience score

94/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs A, Current SAInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Absence of a per-message delivery-status pull endpoint: agent had to inspect the whole OpenAPI spec and stitch labels, inbox events and metrics together after one docs 404.

Where the time went

6:29 agent time
  • Research0:47
  • Attempt4:02
  • Verify1:40
6m 28sTotal elapsed
2:51Time to first success
0:00Waited on the owner

Judge reason

Saved runnable code (agentmail-task/code/run_agentmail.py) calls the live AgentMail REST API with urllib and no mocks; api_transcript.json records 13 real HTTP calls with full bodies. POST /v0/inboxes created the only recipient/sender bench-delivery-20261102064430@agentmail.to (no documented test recipient exists), after sender discovery via GET /v0/auth/me, /v0/domains, /v0/inboxes. Exactly one send: POST /v0/inboxes/{id}/messages/send -> 200 returning message_id <010001a19b009fc7-cffe97a3-4922-43af-8e3a-ccffb5a7ea37-000000@email.amazonses.com>. Pull reads keyed on that id, no webhook/dashboard: GET /v0/inboxes/{id}/messages/{message_id} -> 200 showing state transition labels [sent] at t+0.6s then [sent,received,unread] with timestamp 2026-11-02T06:44:31.757Z / updated_at 06:44:32.110Z; GET /v0/inboxes/{id}/events -> 200 with three label.added events carrying event_at/created_at, client-filtered on message_id; GET /v0/inboxes/{id}/metrics/events?start=2026-11-02T06:44:31Z -> 200 with message.sent/message.delivered count 1, bounced/rejected empty. Transcript line 77 shows the script's live stdout matching the saved result.json byte-for-byte on these values. The report also states the real documented gap (per-message delivered/bounced event objects are push-webhook/Svix-polling only) rather than inventing data.

Attempt record

  1. 1:02Answered from saved key: AgentMail API keyno wait
  2. —First success verified

Attempt artifacts saved.