Benchmark report

Task verified

13m 41s elapsed

The saved attempt met the documented success criterion.

Experience score

91/ 100

Agent-Ready
  • SInstall
  • AAuth
  • SExecute
  • SDocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth A, Execute S, Docs S, Current ASInstallAAuthSExecuteSDocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
Building the no-TCP/WebSocket proof guard took three iterations (stdout broke, sandbox egress proxy on :8080, connect-arg flattening) — self-imposed rigor, not product friction.

Where the time went

13:42 agent time
  • Research0:24
  • Attempt9:59
  • Verify3:18
13m 41sTotal elapsed
5:33Time to first success
0:00Waited on the owner

Judge reason

Verified from saved artifacts and transcript, not prose. Control plane: POST /organizations/vdbs98/projects -> 201 tpcbench-587a4488 (out/06-create-project.json), branch 201 + healthy. HTTP data path: code/xata_http_sql.mjs uses only global fetch (no net/tls/pg/ws imports); out/11-sql-http-guarded.json + stdout show 5x POST https://u30gmsi3gp1dj64t1toc2ktboc.us-east-1.xata.tech/sql all HTTP 200 (DROP/CREATE/INSERT x2/SELECT rowCount=2) with rows [{1,widget,7},{2,gadget,42}] matching printed output; a second real run through installed @neondatabase/serverless 1.2.0 neon() (npm 'added 1 package', lockfile sha512) returned the same rows. No TCP/WebSocket: runs executed under code/no-network-guard.mjs via node --import, which blocks any connect off :443 and makes WebSocket/require('ws') throw; logs 13/14 show only :443 TLS to *.xata.tech via the sandbox CONNECT proxy, exit 0, and an earlier buggy guard version did fail loudly (idx 181), showing the guard is live, not decorative. Cleanup graded: DELETE branch 204, GET branch 404 'not found', list branches [], DELETE project 204, GET project 404, no tpcbench-* in list, and post-delete POST /sql -> 404 BRANCH_NOT_FOUND, each matching transcript tool output. No fabrication: every saved response matches the live transcript results verbatim. Secret hygiene: no API key anywhere; saved files contain only redacted connection strings and a final grep for the literal password/key reported clean. One blemish: the create-branch response was jq-printed once in the session (transcript idx 88) exposing the connection-string password before being moved out of home and redacted, so result.json's 'never printed or saved' overstates slightly; the resource is deleted and no artifact retains it, so this is minor and does not undo the graded criteria.

Attempt record

  1. 1:52Answered from saved key: XATA_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.