Benchmark report

Task verified

13m 43s elapsed

The saved attempt met the documented success criterion.

Experience score

96/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs A, Current SSInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
A 502 for ~3 min: `railway domain --port 3000` set targetPort 3000 while Railway injected PORT=8080, needing an undocumented serviceDomainUpdate fix.

Where the time went

13:43 agent time
  • Research0:23
  • Attempt10:47
  • Verify2:33
13m 43sTotal elapsed
8:05Time to first success
0:00Waited on the owner

Judge reason

Saved CLI/API outputs independently support every criterion. Hard-coded VERSION in work/code/v1|v2/index.js (no env var) with per-request new Date().toISOString(); work/evidence/v1-per-request.txt shows three distinct servedAt values. Deploys via `railway up --ci` to one service; the Railway-assigned URL https://tpcbench-svc-af696951-production.up.railway.app served {"version":"v1",...14:02:33.997Z}, then v2 at 14:03:51.639Z (transcript tool_results, lines 150/170). Rollback used the documented GraphQL mutation deploymentRollback(id) (docs fetch at line 15 shows the page); outputs/14-rollback.json = {"data":{"deploymentRollback":true}}; poll at line 187 captured v2 at 14:04:09 then v1 at 14:04:14.198Z on the same URL. No rebuild: outputs/17 shows the new deployment 96a0c2d5 with meta.reason=rollback and imageDigest sha256:59fcb441... identical to v1 deployment b1f412a6, while v2 was sha256:b4d9d1ad...; outputs/18-buildlogs-rollback.json is {"data":{"buildLogs":[]}}. Cleanup via projectDelete -> true, outputs/21-projects-after.json shows zero projects, and evidence/final-after-cleanup.txt records three 404 'Application not found' fetches. All names use the tpcbench-af696951 suffix; the token was kept in /tmp/priv-railway/curl.cfg outside the home dir and a grep -F scan found it in no saved artifact. (The differing digests in first-success.json are the build-log containerimage.digest values, not contradictory to the deployment meta digests.)

Attempt record

  1. 1:14Answered from saved key: Railway API tokenno wait
  2. —First success verified

Attempt artifacts saved.