A sandbox API for AI agents: practice tool calls against a backend that can't hurt anyone

You're building an agent that calls HTTP APIs — function calling, an MCP toolchain, a LangChain tool, a from-scratch harness. Where does it practice? Pointing it at production is how horror stories start. Pointing it at a read-only toy API teaches it nothing about writes. And the most common shortcut actively lies to your evals:

$ curl -X POST https://jsonplaceholder.typicode.com/posts -d '{"title":"x"}' \
  -H 'content-type: application/json'
{ "title": "x", "id": 101 }        # looks like success…

$ curl https://jsonplaceholder.typicode.com/posts/101
# HTTP 404 — the write never happened

JSONPlaceholder fakes writes by design (fine for frontend demos, its actual job). But an eval that checks "did the agent's POST succeed?" against a faked 201 passes when nothing happened. Your agent learns that fire-and-forget works. It doesn't.

What an agent sandbox actually needs: real state (writes persist, reads see them), a reset button (deterministic episodes), ground truth (a log of what the agent actually sent, not what it claims), and hostile weather (errors, latency, flaky failures — because prod has all three). Mockbird is a free hosted mock API service that happens to have all four. Here's the loop.

1. One curl: a live backend with state

curl -X POST https://mockbird.mockbird.workers.dev/api/projects \
  -H 'content-type: application/json' \
  -d '{"name":"agent sandbox","preset":"ecommerce"}'

Response is {id, adminKey, baseUrl, ...}. No signup. The project comes seeded: 30 products, 25 orders, customers, reviews — a realistic multi-resource world with relations (?_expand=customer works), filtering, pagination, search, and full CRUD that persists. Give your agent the baseUrl and a goal ("find the cheapest product under $200 and create an order for it") and it has a real environment to act in. Keep the adminKey on the harness side — it's the operator's key, not the agent's.

Prefer your own domain shapes? Import an OpenAPI spec, db.json, CSV, or Postman collection and the sandbox mirrors your real API's schema.

2. The episode loop: snapshot → act → assert → restore

Save the pristine state once:

curl -X POST https://mockbird.mockbird.workers.dev/api/projects/PROJECT_ID/snapshots \
  -H "x-admin-key: ADMIN_KEY" -H 'content-type: application/json' \
  -d '{"name":"baseline"}'

Run an episode. Then reset the world to exactly the snapshotted records — every resource, every id, byte for byte:

curl -X POST https://mockbird.mockbird.workers.dev/api/projects/PROJECT_ID/snapshots/baseline/restore \
  -H "x-admin-key: ADMIN_KEY"

Episode N+1 starts from the same state as episode 1. No "previous run left a row behind" flakiness, no teardown scripts. You can keep up to 10 named snapshots per project — a baseline, an empty world, an edge-cases world — and pin read-only scenarios per request with the X-Mockbird-Snapshot header if parallel episodes share one project (details).

Parallel episodes that write: fork, don't share

Snapshot pinning is read-only — the moment two parallel episodes both need to write, a shared project is a race. Fork the template instead: each run gets a byte-identical private copy with its own URL and adminKey, and there is nothing to coordinate and nothing to restore between workers.

# per eval run / CI worker: fork the template project…
FORK=$(curl -s -X POST https://mockbird.mockbird.workers.dev/api/projects/TEMPLATE_ID/fork \
  -H "x-admin-key: TEMPLATE_ADMIN_KEY" -H 'content-type: application/json' \
  -d '{"name":"run-42"}')
RUN_URL=$(echo "$FORK" | jq -r .baseUrl)     # point the agent here
RUN_KEY=$(echo "$FORK" | jq -r .adminKey)    # grader uses this for ground-truth reads
RUN_ID=$(echo "$FORK" | jq -r .id)

# …run the episode against $RUN_URL, grade via read-backs, then throw the world away:
curl -s -X DELETE "https://mockbird.mockbird.workers.dev/api/projects/$RUN_ID" \
  -H "x-admin-key: $RUN_KEY"

The fork copies every record verbatim (ids preserved), custom routes, and settings like protected mode — but not webhooks or the request log, so each run's inspector shows only that run's traffic: cleaner evidence for the grader. Forks count toward normal project limits; delete them when the episode ends.

Or let the fork clean up after itself. Add ?ttl=900 (seconds, 60 s–7 days; also {"ttl": 900} in the body) and the fork auto-deletes when time is up — a harness that crashes mid-run, an eval batch that gets killed, a worker that loses its network: none of them can leak sandboxes, because the cleanup isn't a step your code has to reach. The response carries expiresAt; a run that turns out to be worth keeping is rescued with PUT /settings {"ttl": null} (or extended with a new ttl, always counted from now). Details: /docs#ttl.

# self-cleaning sandbox: no DELETE step to forget
curl -s -X POST 'https://mockbird.mockbird.workers.dev/api/projects/demo/fork?ttl=900'

3. Ground truth: assert on what the agent actually did

Agents confabulate. The transcript says "I created the order" — did it? The request inspector logs the last 50 requests to the project: method, path, query, status, headers, body snippet.

curl https://mockbird.mockbird.workers.dev/api/projects/PROJECT_ID/requests \
  -H "x-admin-key: ADMIN_KEY"

{ "ts": 1788800332775, "method": "POST", "path": "/orders", "status": 201,
  "body": "{\"status\":\"pending\",\"total\":42.5}",
  "headers": { "content-type": "application/json", ... } }

Your eval can check both layers: state (GET the record the agent claims to have created) and behavior (did it send one POST or five? did it set the right content-type? did it retry the 500 or give up?). Behavior assertions are where agent bugs hide — a harness that only checks final state misses the agent that succeeded by accident.

Trajectory assertions: grade the how, not just the what

The inspector takes filters and returns count — matches in the retained window — so behavior assertions are one-liners, no client-side log parsing:

# the agent never deleted anything (destructive-action constraint):
curl -s '.../api/projects/FORK_ID/requests?method=DELETE' -H "x-admin-key: KEY" \
  | jq -e '.count == 0'

# it stayed inside /tasks (scope constraint — no rummaging in /users):
curl -s '.../api/projects/FORK_ID/requests?path=/users' -H "x-admin-key: KEY" \
  | jq -e '.count == 0'

# it made exactly one write, and no request errored (efficiency + hygiene):
curl -s '.../api/projects/FORK_ID/requests?method=POST&path=/orders' -H "x-admin-key: KEY" \
  | jq -e '.count == 1'
curl -s '.../api/projects/FORK_ID/requests?status_gte=400' -H "x-admin-key: KEY" \
  | jq -e '.count == 0'

Available filters: method= (comma-list), path= (segment-aware prefix), status=, status_gte=/status_lte=, and since= (epoch ms or ISO-8601 — record the episode start, assert about only that episode). This composes with the fork-per-run pattern from §2½: a fresh fork's log contains exactly one episode's traffic, so "the last 50 requests" is never a limitation — it's the whole trace. A state diff that passes plus a trajectory that violated constraints is a run your rubric should fail; now both checks are single jq -e exit codes. Same filters over MCP (inspect_requests with method/path/… args) and on read-only share links (/api/share/<token>/requests?method=DELETE) so a human reviewer can check the same constraints without the admin key.

Need the trace in a form humans already have tooling for? GET /api/projects/FORK_ID/requests.har exports the same window (same filters) as a standard HAR 1.2 file — attach it to a bug report or open it in Chrome/Firefox DevTools (Network tab → Import HAR) to scrub through the episode request by request. Responses are status-only (the inspector records what the agent sent); share-link holders get it keylessly at /api/share/<token>/requests.har. See docs.

Scoring rule: the response body earns no credit

The single most common false positive in agent evals: grading the response the API sent back. A convincing 201 body is not evidence of persistence — JSONPlaceholder returns {"id": 101} for a POST it never saved, and a GET of /posts/101 404s a second later. An eval that trusts the body passes an agent on an API that lied to both of them.

Grade the read-back instead. After the episode ends, the harness — not the agent — fetches the record through an independent request:

# agent claims it created an order; the grader checks for itself
curl https://mockbird.mockbird.workers.dev/m/PROJECT_ID/orders/CLAIMED_ID
# 200 with the right fields -> credit. 404 -> no credit, whatever the transcript says.

Three lines for your rubric: (1) the agent's transcript is a claim, not evidence; (2) the API's response body is a claim, not evidence; (3) only an independent read-back of state — plus the inspector log of what was actually sent — is evidence. On a mock with real persistence all three are cheap to collect.

Read-after-write: the guarantee your grader is standing on

Read-back grading only works if a missing record is evidence — not replication lag. If your sandbox is eventually consistent, a grader that GETs immediately after the episode has to poll-with-timeout, and every retry window is a place where a real agent failure can hide behind "maybe it just hasn't propagated yet".

Mockbird's guarantee is the strong one: every write is committed to the database before the 2xx is returned. No write queue, no async replication, no read replicas. The 201 means the row exists; a GET issued the next millisecond returns it, and a delete reads back 404 immediately:

# the whole sequence is deterministic — no sleeps, no retries
curl -X POST https://mockbird.mockbird.workers.dev/m/demo/products \
  -H "content-type: application/json" -d '{"name":"RAFW probe","price":1.23}'
# -> 201 {"name":"RAFW probe","price":1.23,"id":34}
curl https://mockbird.mockbird.workers.dev/m/demo/products/34   # -> 200, same fields
curl -X DELETE https://mockbird.mockbird.workers.dev/m/demo/products/34
curl https://mockbird.mockbird.workers.dev/m/demo/products/34   # -> 404, instantly

So your grader can be strict with no flake budget: award credit only when the 2xx is followed by an independent GET whose fields match the intended write (and, for deletes, a read-back 404). A miss is a real failure — the agent wrote the wrong thing, the wrong place, or nothing at all — never a timing artifact. The only asynchronous side effect in Mockbird is outbound webhook delivery; record state itself is never deferred.

Snapshot diff: the whole-state verdict in one call

Read-back grading checks the records the agent mentioned. It misses collateral damage: the agent that completed the task but also deleted three unrelated records, or edited a field nobody asked it to touch. The snapshot diff endpoint compares a saved snapshot (expected) against the project's live data (actual) and returns a structured verdict for every record in every resource:

curl "https://mockbird.mockbird.workers.dev/api/projects/PROJECT_ID/snapshots/expected/diff" \
  -H "x-admin-key: ADMIN_KEY"

{ "identical": false,
  "summary": { "resourcesAdded": [], "resourcesRemoved": [],
               "records": { "added": 1, "removed": 1, "changed": 1, "unchanged": 22 } },
  "resources": [ { "resource": "tasks",
    "added": [6], "removed": [2],
    "changed": [ { "id": 3, "fields": { "done": { "expected": false, "actual": true } } } ],
    "unchanged": 2 } ] }

added = records that exist now but weren't in the snapshot; removed = were in the snapshot, gone now; changed shows per-field expected-vs-actual pairs. Comparison is canonical (key order never causes a false diff), and ?ignore=updatedAt,createdAt excludes volatile fields everywhere. The verdict also rides the x-mockbird-identical: true|false response header, so a harness can grade with curl -s … | jq -e '.identical' and an exit code.

Two grading patterns:

# A. Answer-key grading: author the correct end-state, snapshot it, then
#    reset and let the agent try. Grader = one GET + one boolean.
#    You can WRITE the answer key inline — no need to act it out on live data first:
curl -X POST .../api/projects/PID/snapshots -H "x-admin-key: KEY" -H "content-type: application/json" \
  -d '{"name":"expected","data":{"tasks":[{"id":1,"title":"write tests","done":true},{"id":2,"title":"ship it","done":true}]}}'
curl -X POST .../api/projects/PID/snapshots/start/restore -H "x-admin-key: KEY"              # reset to the task's start state
# ... agent episode runs ...
curl -s ".../api/projects/PID/snapshots/expected/diff?ignore=updatedAt" -H "x-admin-key: KEY" | jq -e '.identical'

# B. Change-set grading (parallel runs): fork per episode, carrying the template's
#    snapshots, then require the diff vs "start" to be EXACTLY the intended change.
curl -X POST .../api/projects/TEMPLATE/fork -H "x-admin-key: KEY" \
  -H "content-type: application/json" -d '{"name":"run-42","withSnapshots":true}'
# ... agent episode runs against the fork ...
# grader asserts: diff vs start = exactly one added order, nothing else touched
curl -s ".../api/projects/FORK_ID/snapshots/start/diff" -H "x-admin-key: FORK_KEY" \
  | jq -e '.summary.records == {"added":1,"removed":0,"changed":0,"unchanged":24}'

Pattern B is the one that catches "succeeded, but trashed the place on the way out" — the transcript never confesses to that; the diff always does.

Authored snapshots (the inline data form in pattern A) store your records verbatim — ids preserved, [] means "expected: this resource emptied", and new resource names are allowed (restore rebuilds them). Details: docs § authored snapshots.

One call to grade it all: the verdict endpoint

Once you're grading both state (diff) and behavior (trajectory), the grader is several curls and some jq glue. The verdict endpoint folds them into one call, one boolean:

curl -s https://mockbird.mockbird.workers.dev/api/projects/FORK_ID/verdict \
  -H "x-admin-key: FORK_KEY" -H "content-type: application/json" -d '{
    "snapshot": "expected", "ignore": ["updatedAt"],
    "trajectory": [
      {"method": "DELETE", "count": 0},
      {"method": "POST", "path": "/orders", "count": 1},
      {"status_gte": 400, "count": 0}
    ],
    "failStatus": 422
  }' | jq -e '.pass'

The response is {pass, checks[]} — the state check carries the full diff breakdown, each trajectory check carries its matched count, expectation, and a detail string when it fails. Each constraint takes the inspector filters (method, path, status, status_gte/lte, since) plus an expectation: count (exact), min and/or max. Both halves are optional — state-only or behavior-only verdicts work.

failStatus is the CI-friendly trick: a failing verdict returns that HTTP status instead of 200, so curl -sf — or any HTTP client's error path — is the gate, no JSON parsing at all. Same call over MCP: the verdict tool ({"snapshot":"expected","trajectory":[...]} → assert .pass). Reference: docs § verdict.

Saved verdicts: the grader never sees the spec (or the key)

One wrinkle remains in that harness: whoever runs the verdict holds the admin key and the answer key. Fix both by saving the spec by name on the template — forks copy it — and grading through the share link:

# once, on the template: author the grading spec
curl -X PUT https://HOST/api/projects/TEMPLATE/verdicts/final -H 'x-admin-key: TKEY' \
  -H 'content-type: application/json' \
  -d '{"snapshot":"expected","trajectory":[{"status_gte":400,"count":0}],"failStatus":422}'

# per run: fork (copies saved verdicts; withSnapshots carries "expected"), mint a share link
# ... agent works the fork ...

# the grader/CI runs the check with ONLY the share token + the name:
curl -sf https://HOST/api/share/<token>/verdict/final

The agent under test gets the fork's /m URL and can't read the spec (it lives behind the admin API); the grader gets a link that can only read — running a check mutates nothing. With failStatus, that one keyless curl -sf is the entire CI gate. Humans get the same thing as one-click Run buttons on the share page. Docs →

And for the humans who never open a terminal: append /badge.svg to that same URL and you get a live pass/fail badge — embed it in the eval run's PR or a results README and every render re-grades the run against the live sandbox:

![final](https://HOST/api/share/<token>/verdict/final/badge.svg)

4. Hostile weather: does your agent survive a flaky API?

Real APIs return 500s, rate-limit, and hang. An agent that has only ever seen 200s has untested error paths in every tool call. Every Mockbird endpoint takes failure-injection params — no config, just the URL:

# always fail: exact error handling
curl -i "https://mockbird.mockbird.workers.dev/m/demo/products?mock_status=500"

# fail ~half the time with realistic statuses (500/502/503/504/429) — retry training
curl "https://mockbird.mockbird.workers.dev/m/demo/products?mock_chaos=0.5"

# 2s latency: timeout and patience handling
curl "https://mockbird.mockbird.workers.dev/m/demo/products?mock_delay=2000"

Chaos-failed writes are not applied (server-died-before-processing semantics), so "retry until the write lands, then verify" is exactly the behavior a correct agent must learn — and the inspector shows whether yours did. Injected failures carry an x-mockbird-chaos: injected header so your harness can tell scripted weather from real bugs. Recipes: error-state testing, rate-limit simulation.

5. Auth practice: tokens without a real identity provider

Agents also have to learn the login → token → authorized-request dance. The sandbox has a real one: POST /auth/login with any email/password returns a signed JWT (configurable expiry — set it to 5 seconds to drill token-refresh handling), and flipping the project to protected mode makes every endpoint return a proper 401 without a Bearer token. Try it on the demo right now:

curl -X POST https://mockbird.mockbird.workers.dev/m/demo/auth/login \
  -H 'content-type: application/json' \
  -d '{"email":"agent@example.com","password":"hunter2"}'

Full walkthrough: mock JWT auth.

6. If your agent speaks MCP

Everything above is plain HTTP — any agent that can make a request can use it. If your agent runs in an MCP client (Claude Code, Cursor, VS Code, or your own), Mockbird also ships a hosted MCP server at https://mockbird.mockbird.workers.dev/mcp — the agent can create its own sandbox, query and write records, save/restore snapshots, and read the inspector as first-class tools instead of raw curl. The whole harness above works over MCP too: fork_project (with ttl for self-expiring forks and withSnapshots to carry the answer key) plus snapshots action:"diff" for the verdict. Setup is one config line: the MCP guide.

7. Let a human review the run — without handing over the key

When the run is done (or while it's still going), the agent can mint a read-only share link and hand it to whoever supervises it:

curl -X POST https://HOST/api/projects/PROJECT/share -H 'x-admin-key: KEY'
# → {"shareUrl": "https://HOST/share/<token>"}

The link opens a live browser view of the sandbox — every resource with its records, custom routes, snapshots, exports, and the request inspector showing exactly what the agent called and when. The holder can watch but not write, and the admin key never appears; rotate or revoke the link any time. Over MCP it's the share_project tool. This closes the loop for supervised runs: the agent does the work, the human reads the evidence, nobody copies secrets around. Docs →

Honest comparison

Sandbox option	Writes persist	Reset between episodes	Action log	Failure injection	Setup
JSONPlaceholder / FakeStoreAPI	❌ faked (evals get false positives)	n/a (read-only)	❌	❌	none
httpbin-style echo	❌ stateless	n/a	❌	partial (status/delay)	none
Local WireMock / Express stub	✔ if you build it	you write it	✔ (WireMock verification)	✔ (WireMock; Express DIY)	process per env, no hosted URL for cloud agents
Real staging environment	✔	slow, often shared	maybe (APM)	hard to do on purpose	exists already, but agents can break it
Mockbird	✔	✔ snapshot restore	✔ inspector (last 50)	✔ status/chaos/delay/jitter	one curl, hosted URL

Where the others win, plainly: WireMock's request verification DSL is richer than our inspector (count matchers, ordering); a local stub is offline and has no request cap; a real staging env exercises your actual business logic, which no mock can. Use Mockbird for the tool-calling mechanics — HTTP, auth, retries, CRUD semantics — and staging for the last mile.

Limits, honestly: 10,000 requests/project/day (plenty for eval suites, not for load tests), 50-request inspector window (poll it between episodes if you need full traces), 10 snapshots/project, 1,000 records/resource, 30 anonymous projects per IP per day — a big fleet behind one egress IP will hit that; spread projects or reuse one with snapshots. Data is mock-grade: don't store anything you care about.

More guides: MCP mock API server · deterministic test data · simulating rate limits · testing loading & error states. Create your API →