Mock the OpenAI API — test AI clients and agents without burning tokens

Your test suite calls the OpenAI API. That means every CI run costs money, needs a real key in a secret store, returns something different every time (good luck asserting on it), and occasionally eats a 429 that has nothing to do with your code. The usual fixes are in-process stubs (respx, nock — invisible to subprocesses, other services, and anything that isn't your test runner) or running a fake-OpenAI container yourself.

Mockbird gives you a hosted OpenAI-compatible mock instead: real HTTPS URL, deterministic replies, streaming that actually speaks SSE, and one-query-param failure injection. Point OPENAI_BASE_URL at it and your code doesn't change at all. Everything below was verified with the official openai Python SDK (v2.x) against the commands shown.

To be clear about what this is: there is no model behind it. It returns the deterministic responses you define — which is exactly what you want in tests. You're testing your parsing, streaming, retry, timeout and fallback logic, not the model.

1. Create the mock (one paste)

BASE=https://mockbird.mockbird.workers.dev
P=$(curl -s -X POST $BASE/api/projects -H 'content-type: application/json' \
     -d '{"name":"openai-mock","blank":true}')
PID=$(echo $P | sed 's/.*"id": *"\([^"]*\)".*/\1/')
KEY=$(echo $P | sed 's/.*"adminKey": *"\([^"]*\)".*/\1/')

# chat completions (JSON) — echoes the model + your messages back
curl -s -X POST $BASE/api/projects/$PID/routes -H "x-admin-key: $KEY" \
  -H 'content-type: application/json' -d '{
  "method":"POST","path":"/v1/chat/completions","status":200,
  "body":"{\"id\":\"chatcmpl-{{uuid}}\",\"object\":\"chat.completion\",\"created\":{{{ts}}},\"model\":\"{{body.model}}\",\"choices\":[{\"index\":0,\"message\":{\"role\":\"assistant\",\"content\":\"Hello from your Mockbird mock! This reply is deterministic - assert on it.\"},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":57,\"completion_tokens\":17,\"total_tokens\":74},\"mockbird_echo\":{{{body.messages}}}}"
}'

# models list
curl -s -X POST $BASE/api/projects/$PID/routes -H "x-admin-key: $KEY" \
  -H 'content-type: application/json' -d '{
  "method":"GET","path":"/v1/models","status":200,
  "body":"{\"object\":\"list\",\"data\":[{\"id\":\"mock-gpt\",\"object\":\"model\",\"created\":1722600000,\"owned_by\":\"mockbird\"}]}"
}'

echo "base_url: $BASE/m/$PID/v1"

Your mock's base URL is https://mockbird.mockbird.workers.dev/m/<PID>/v1. The {{body.model}} and {{{body.messages}}} bits are response templates: the completion echoes back whatever model you asked for, and a mockbird_echo field carries your exact messages array — so a test can assert the system prompt your agent actually sent.

2. Point the SDK at it — no code change

export OPENAI_BASE_URL="https://mockbird.mockbird.workers.dev/m/<PID>/v1"
export OPENAI_API_KEY="mock-key"   # any non-empty string; the mock doesn't check it

from openai import OpenAI

client = OpenAI()  # reads both env vars — your production code, untouched
r = client.chat.completions.create(model="gpt-4o-mini",
    messages=[{"role": "system", "content": "You are a test."},
              {"role": "user", "content": "ping"}])

assert r.choices[0].message.content.startswith("Hello from your Mockbird mock!")
assert r.model == "gpt-4o-mini"          # echoed from the request
assert r.usage.total_tokens == 74        # deterministic
assert r.model_extra["mockbird_echo"][1]["content"] == "ping"   # the prompt arrived intact

The response parses into the SDK's typed ChatCompletion — id, choices, finish_reason, usage all present. The Node SDK works the same way (baseURL option or the same env vars), as does anything OpenAI-compatible: LangChain, LiteLLM, the Vercel AI SDK's OpenAI provider.

3. Streaming (real SSE, parsed by the real SDK)

Streaming clients need text/event-stream with chat.completion.chunk frames and a data: [DONE] terminator. Define a second route for your streaming path (a custom route serves one fixed body, so give streaming its own prefix):

curl -s -X POST $BASE/api/projects/$PID/routes -H "x-admin-key: $KEY" \
  -H 'content-type: application/json' -d '{
  "method":"POST","path":"/stream/v1/chat/completions","status":200,
  "contentType":"text/event-stream",
  "body":"data: {\"id\":\"chatcmpl-mock1\",\"object\":\"chat.completion.chunk\",\"created\":1722600000,\"model\":\"mock-gpt\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-mock1\",\"object\":\"chat.completion.chunk\",\"created\":1722600000,\"model\":\"mock-gpt\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Hello from your mock!\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-mock1\",\"object\":\"chat.completion.chunk\",\"created\":1722600000,\"model\":\"mock-gpt\",\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}]}\n\ndata: [DONE]\n\n"
}'

stream_client = OpenAI(base_url=f"{BASE}/m/{PID}/stream/v1", api_key="mock-key")
parts = []
for chunk in stream_client.chat.completions.create(model="mock-gpt", stream=True,
        messages=[{"role": "user", "content": "ping"}]):
    if chunk.choices[0].delta.content:
        parts.append(chunk.choices[0].delta.content)
print("".join(parts))   # assembled exactly as your UI would render it

The SDK's stream iterator consumes it chunk by chunk and stops at [DONE] — your token-by-token rendering, "stop button", and stream-assembly code all get exercised. One honest caveat: the whole body arrives at once rather than paced per token. To test slow streams, add ?mock_delay=2000 for time-to-first-token, or point EventSource-style readers at our paced SSE endpoint.

4. Embeddings with real 1536-dim vectors

A route body can hold a full 1536-dimension vector (16 KB template budget — a 4-decimal vector fits with room to spare), so code that asserts len(embedding) == 1536 passes unmodified. Generate the route once:

python3 - <<'EOF'
import json, math, os, urllib.request
BASE, PID, KEY = "https://mockbird.mockbird.workers.dev", os.environ["PID"], os.environ["KEY"]
vec = [round(math.sin(i) * 0.05, 4) for i in range(1536)]   # deterministic, non-degenerate
body = ('{"object":"list","data":[{"object":"embedding","index":0,"embedding":'
        + json.dumps(vec) + '}],"model":"{{body.model}}","usage":{"prompt_tokens":8,"total_tokens":8}}')
req = urllib.request.Request(f"{BASE}/api/projects/{PID}/routes",
    data=json.dumps({"method": "POST", "path": "/v1/embeddings", "body": body}).encode(),
    headers={"content-type": "application/json", "x-admin-key": KEY, "user-agent": "setup"})
urllib.request.urlopen(req)
EOF

e = client.embeddings.create(model="text-embedding-3-small", input="hello world")
assert len(e.data[0].embedding) == 1536   # passes — it's a real vector

5. Failure drills — the part you can't test against the real API

Every simulation parameter works on custom routes, and the SDK lets you attach query params to every request via default_query:

Scenario	Client setup	What happens
Hard 500	`default_query={"mock_status": "500"}`	SDK raises `InternalServerError` — test your fallback path
Rate limit, then recovery	`default_query={"mock_seq": "429,200"}`	1st request 429, 2nd succeeds — the SDK's auto-retry recovers; assert it did
Flaky API	`default_query={"mock_chaos": "0.3"}`	30% of requests fail with 500/502/503/504/429 at random
Slow model	`default_query={"mock_delay": "3000"}`	3 s latency — test spinners and client timeouts
Real rate-limit headers	`default_query={"mock_ratelimit": "5"}`	5 req/min then 429 + `Retry-After` + `x-ratelimit-*` countdown headers

retry_client = OpenAI(base_url=MOCK, api_key="mock-key",
    default_query={"mock_seq": "429,200"}, max_retries=2)
r = retry_client.chat.completions.create(model="m", messages=[...])
# succeeds — but only because the SDK retried the injected 429. Deterministic, every run.

mock_seq is the one to reach for in CI: unlike random chaos it serves an exact status script (503,503,200 = fail twice, then succeed), isolated per test via mock_seq_key. See testing loading & error states for the full recipes.

6. See what your agent actually sent

Every request lands in the project's request inspector — method, path, headers, and the request body. When an agent misbehaves, the first question is "what prompt did it actually send?" — and the answer is sitting in your dashboard:

curl -s $BASE/api/projects/$PID/requests -H "x-admin-key: $KEY"
# … {"method":"POST","path":"/v1/chat/completions","status":200,
#    "body":"{\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}],\"model\":\"gpt-4o-mini\"}"} …

(Bodies are captured up to 2 KB — enough for most prompts; long RAG contexts get truncated.)

Honest comparison

	In-process stubs (respx / nock / VCR)	Self-hosted fake-OpenAI container	Mockbird
Reachable by subprocesses, other services, CI, teammates	✗ — same-process only	✔ on your network	✔ public HTTPS
Setup	per-test-runner code	Docker + hosting	two curls
Streaming SSE	varies, often hand-rolled	✔	✔ (verified with the real SDK)
Failure/latency injection	you write it	varies	query params (429/500/chaos/seq/delay/rate-limit)
Offline / air-gapped	✔	✔	✗ — it's hosted
Token-paced streaming	you write it	some do it	✗ — body arrives at once

If your tests must run offline, an in-process stub is the right call. The moment the code under test is an agent spawning subprocesses, a service in a docker-compose stack, a teammate's notebook, or CI hitting a deployed preview — an in-process stub can't see any of it, and a hosted URL can.

Notes & limits

Anything OpenAI-compatible works against it: the official Python/Node SDKs, LangChain, LiteLLM proxies, Vercel AI SDK. Same trick mocks Anthropic-style /v1/messages or any other JSON AI API — it's just custom routes.
The mock never checks your API key — never send a real one anywhere near test config anyway.
Responses are the fixed bodies you defined (with per-request template fields like {{uuid}}). Need several canned replies? Define more routes (/v1a/…, /v1b/…) and switch base_url per test, or script statuses with mock_seq.
Free tier: 10,000 requests/project/day, 20 custom routes, 16 KB per response template. It's a mock — don't point production at it.

Create yours

The one-paste block in section 1 is the whole setup — or start from the dashboard and add the routes in the UI. Mocking the REST backend your agent talks to as well? The same project serves a full CRUD mock API, GraphQL, and webhooks. Agents can even set all of this up themselves over MCP.