Benchmark report

Task verified

8m 11s elapsed

The saved attempt met the documented success criterion.

Experience score

94/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs A, Current SAInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Polling GET /events 12 times over ~2 minutes for a 'delivered'/'bounced' label event that AgentMail never emits per message; only aggregate metrics show delivery.

Where the time went

8:11 agent time
  • Research0:32
  • Attempt5:48
  • Verify1:51
8m 11sTotal elapsed
4:29Time to first success
0:00Waited on the owner

Judge reason

Saved code (agentmail-run/agentmail_client.py, run_send_and_read_events.py, cleanup_and_final_reads.py) ran live against https://api.agentmail.to with a key obtained via benchmark-request (outcome: provided). Logged calls: POST /v0/inboxes x2 created sender evt-sender-20261109191838@agentmail.to and recipient evt-rcpt-20261109191838@agentmail.to (inbox fallback, since AgentMail documents no test recipient); POST /v0/inboxes/{sender}/messages/send -> 200 with to=evt-rcpt-... in the saved request body and message_id <010001a1c1bf915b-4479710e-fa6e-4e75-9301-f488b6f873c1-000000@email.amazonses.com>. Pull-style read GET /v0/inboxes/{sender}/events?limit=100 -> 200 returned event_id 01M70VZ3MRR3WNZXTXAX9E0000, event_type label.added, label 'sent', event_at 2026-11-09T19:18:39.000Z, created_at 2026-11-09T19:18:40.214Z, filtered client-side on that message_id (no webhook/dashboard). Corroborated by GET .../messages/{message_id} (labels ['sent'], timestamp 2026-11-09T19:18:39.642Z), GET .../metrics/events (message.sent 1, message.delivered 1, message.bounced []), and recipient-inbox events (received/unread for the same message_id). Transcript tool output at the live run matches the saved JSON logs verbatim. Recipient inbox deleted: DELETE /v0/inboxes/evt-rcpt-... -> 202 empty body (sender inbox also deleted, 202), recorded in result.json. result.json honestly states the documented gap (no message_id filter on metrics, no per-message delivery-status field) rather than inventing a delivered/bounced event.

Attempt record

  1. 1:39Answered from saved key: AgentMail API key (AGENTMAIL_API_KEY, am_...)no wait
  2. —First success verified

Attempt artifacts saved.