Benchmark report
Task verified
10m 9s elapsed
The saved attempt met the documented success criterion.
66/ 100
Needs Alignment- SInstall
- BAuth
- AExecute
- BDocs
- ACurrent
Hunting for a routine_id: docs publish none, and both documented discovery paths (CLI, MCP) rejected the API key with 403/401 requiring interactive login.
Where the time went
10:09 agent time- Research0:25
- Attempt7:27
- Verify2:17
Judge reason
Saved code (code/clay.py + code/enrich_stripe.py) makes real urllib HTTPS calls to Clay's Public API with the documented clay-api-key header. evidence/01-create-search.json shows POST /public/v0/search/query-mode HTTP 200 (search_id=search_0tol8ss5FzM9QmZvc47) with live CloudFront/Request-Id headers; evidence/02-company-results.json shows POST /search/query-mode/{id}/run HTTP 200 returning one record: name 'Stripe', domain 'stripe.com', industry 'Technology, Information and Internet', size '5,001-10,000', linkedin_url .../company/stripe. The body matches the documented PublicApiCompanySearchResult / NextQueryModeCompanyResultsResponse schema in docs/openapi.json (all 12 required fields present), and the company was identified by domain via the documented clay.include_company_identifiers(("stripe.com")) query confirmed in evidence/query-reference.json. evidence/run-output.txt printed values match the saved response exactly. Nothing mocked. The planned Routines enrichment path was unusable because Clay publishes no routine_id listing (GET /public/v0/routines 404; CLI v3 workspaces/tools 403 'Insufficient permissions'; MCP tools/list 401 'You must be logged in' — evidence/03-routine-discovery-blocked.json), so the agent used another officially documented, key-authenticated operation meeting the same criteria rather than guessing an endpoint. No API key, access token or bearer token appears in any saved file or printed output (independently grepped; agent also grepped with the live key value and found no match).
Attempt record
- 1:53Answered from saved key: Clay API keyno wait
- —First success verified
Attempt artifacts saved.