Benchmark report

Task verified

8m 58s elapsed

The saved attempt met the documented success criterion.

Experience score

99/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • SDocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs S, Current ASInstallSAuthSExecuteSDocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
The pooler kept serving stale auth for the dropped role, so proving the revoked credential fails took ~2.5 minutes of repeated connect attempts.

Where the time went

8:58 agent time
  • Research0:23
  • Attempt6:41
  • Verify1:53
8m 58sTotal elapsed
2:40Time to first success
0:00Waited on the owner

Judge reason

Saved code+output prove the full lifecycle on live Supabase. POST /v1/projects -> HTTP 201 ref jalbiisndrylzuzhkapg name tpcbench-8bc77bd8 (03-create-project.json); pooler host from GET /config/database/pooler (05). Admin connect over aws-0-us-east-1.pooler.supabase.com:5432 sslmode=require, \conninfo shows TLSv1.3/TLS_AES_256_GCM_SHA384 (12,13). Scoped credential made by the documented SQL path (supabase.com/docs/guides/database/postgres/roles): CREATE ROLE tpcbench_reader_8bc77bd8 WITH LOGIN PASSWORD + GRANT USAGE/SELECT; pg_roles shows rolsuper=f and role_table_grants shows SELECT only (14). Connecting as tpcbench_reader_8bc77bd8.jalbiisndrylzuzhkapg with sslmode=require: TLS 1.3 conninfo, current_user=tpcbench_reader_8bc77bd8, SELECT returns 3 rows, INSERT -> 'ERROR: permission denied for table items' exit 1 (15). REVOKE+DROP ROLE with follow-up reads showing 0 role rows and 0 grants (16); later connects fail - invalid role OID, then FATAL (EAUTHQUERY) user not found, then FATAL password authentication failed (17-19). DELETE /v1/projects/{ref} -> HTTP 200, GET -> 404 'Resource has been removed', project list shows no tpcbench-* and admin connect gives FATAL (ENOTFOUND) tenant/user not found (20-22). Independent grep of /workspace/home found no password or password-bearing connection string (only ${VAR} refs and doc placeholders). Management API role endpoints were skipped as Beta per the saved OpenAPI excerpts (07-09), and the SQL path the task explicitly permits was used instead.

Attempt record

  1. 0:47Answered from saved key: SUPABASE_ACCESS_TOKENno wait
  2. —First success verified

Attempt artifacts saved.