Drift Detection

A mock that does not match reality is worse than no mock. llmock includes three-way drift tests that compare SDK types, real API responses, and mock output to catch shape mismatches before your users do.

Three-Way Comparison

Each drift test compares three sources:

{ }

SDK Types

What TypeScript types say the shape should be

⇄

Real API

What OpenAI, Claude, Gemini actually return

⚙

llmock

What the mock produces for the same request

Mock doesn't match real

llmock needs updating — test fails immediately. The SDK comparison tells us why it drifted.

Provider changed, SDK is behind

Early warning — the real API has new fields that neither the SDK nor llmock know about yet.

All three agree

No drift — the mock matches reality and the SDK types are current.

Running Drift Tests

Run drift tests shell

# Set API keys for providers you want to test
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GOOGLE_API_KEY=AI...

# Run all drift tests
pnpm test:drift

# Run for a specific provider
pnpm test:drift -- --grep "OpenAI Chat"

Test Files

File	Provider	What it tests
openai-chat.drift.ts	OpenAI	Chat Completions (streaming + non-streaming, text + tool calls)
openai-responses.drift.ts	OpenAI	Responses API (HTTP SSE)
anthropic.drift.ts	Anthropic	Claude Messages API
gemini.drift.ts	Google	Gemini generateContent + streamGenerateContent
ws-realtime.drift.ts	OpenAI	Realtime API over WebSocket
ws-responses.drift.ts	OpenAI	Responses API over WebSocket
ws-gemini-live.drift.ts	Google	Gemini Live over WebSocket
models.drift.ts	All	Model list endpoint conformance

How Drift Analysis Works

drift-test.ts ts

import { extractShape, triangulate, formatDriftReport, shouldFail } from "./schema";

// 1. Get the SDK shape (what TypeScript says)
const sdkShape = openaiChatCompletionShape();

// 2. Call the real API and the mock in parallel
const [realRes, mockRes] = await Promise.all([
  openaiChatNonStreaming(config, [{ role: "user", content: "Say hello" }]),
  httpPost(`${instance.url}/v1/chat/completions`, { /* ... */ }),
]);

// 3. Extract response shapes
const realShape = extractShape(realRes.body);
const mockShape = extractShape(JSON.parse(mockRes.body));

// 4. Three-way comparison
const diffs = triangulate(sdkShape, realShape, mockShape);
const report = formatDriftReport("OpenAI Chat (non-streaming text)", diffs);

// 5. Critical diffs fail the test
if (shouldFail(diffs)) {
  expect.soft([], report).toEqual(
    diffs.filter(d => d.severity === "critical")
  );
}

Severity Levels

Severity	Meaning	Action
critical	Mock does not match real API	Test fails. llmock needs updating.
warning	Provider added new field, neither SDK nor mock have it	Logged. Early warning for future breakage.
ok	All three agree	No action needed.

CI Integration

Drift tests run daily in CI with real API keys stored as GitHub secrets. Tests that require API keys are automatically skipped when the key is not set, so pnpm test:drift is safe to run locally without any keys configured.

Drift tests require real API keys and make real API calls. They are not part of the regular pnpm test suite and must be run explicitly with pnpm test:drift.