Benchmark report

Task verified

8m 26s elapsed

The saved attempt met the documented success criterion.

Experience score

93/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • ADocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs A, Current AAInstallSAuthSExecuteADocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
Polling (12 + 6 times) for a per-message delivered/bounced label AgentMail never attaches; delivery state exists only as an aggregate metrics count.

Where the time went

8:26 agent time
  • Research0:24
  • Attempt6:26
  • Verify1:36
8m 26sTotal elapsed
3:18Time to first success
0:00Waited on the owner

Judge reason

Saved evidence (agentmail-run/evidence/api_calls.json, 19 real HTTP calls) shows a live run against https://api.agentmail.to: GET /v0/auth/me 200 (pod-scoped key), GET /v0/inboxes 200 discovering permitted sender bench-sender-1793605598@agentmail.to, POST /v0/inboxes 200 creating recipient bench-recipient-1794249740@agentmail.to for the run, then exactly one POST /v0/inboxes/{sender}/messages/send 200 returning message_id <010001a1c19e5413-...@email.amazonses.com>. Pull reads follow: GET /v0/inboxes/{sender}/events (documented 'List label change events') returned event_id 01M70SWMP82DD704QE5T1K0000, event_type label.added, label 'sent', event_at 2026-11-09T18:42:21.000Z for that exact message_id; GET /v0/inboxes/{sender}/messages/{message_id} returned labels ['sent'], timestamp 2026-11-09T18:42:21.261Z; GET .../messages?subject=<this subject> returned count 1 with the same id. A later followup_poll.py added GET /v0/inboxes/{sender}/metrics/events?start=... 200 showing message.sent 1 and message.delivered 1 at 2026-11-09T18:45:44.862Z, message.bounced []. No webhook or dashboard was used; code is plain urllib with no mocking or hard-coded responses, and REPORT.md's tables match the saved JSON byte for byte. Cleanup is present: DELETE /v0/inboxes/bench-recipient-1794249740@agentmail.to -> HTTP 202 (empty body), recorded in api_calls.json and result.json. The report also states the real API limitation honestly: per-message delivered/bounced is only exposed via webhooks, so the pull path yields the 'sent' label event plus the aggregate metrics feed.

Attempt record

  1. 1:42Answered from saved key: AGENTMAIL_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.