Chaos Testing

llmock provides probabilistic failure injection to test how your application handles unreliable LLM APIs. Three failure modes can be configured at the server, fixture, or per-request level.

Failure Modes

Mode	Action	Description
`drop`	HTTP 500	Returns a 500 error with `{"error":{"message":"Chaos: request dropped","code":"chaos_drop"}}`
`malformed`	Broken JSON	Returns HTTP 200 with invalid JSON body: `{malformed json: <<<chaos>>>`
`disconnect`	Connection destroyed	Destroys the TCP connection immediately with no response

Precedence

Chaos configuration is resolved with a three-level precedence hierarchy. Higher levels override lower ones:

Per-request headers (highest) — override everything
Fixture-level config — overrides server defaults
Server-level defaults (lowest)

Within a single level, modes are evaluated in order: drop, malformed, disconnect. The first mode that triggers (based on its probability) wins.

Quick Start

chaos-quick-start.ts ts

import { LLMock } from "@copilotkit/llmock";

const mock = new LLMock();
mock.onMessage("hello", { content: "Hi!" });

// 50% of all requests will be dropped with a 500
mock.setChaos({ dropRate: 0.5 });

await mock.start();

// Later, remove chaos
mock.clearChaos();

Programmatic API

Programmatic chaos control ts

// Set server-level chaos (returns `this` for chaining)
mock.setChaos({
  dropRate: 0.1,        // 10% drop rate
  malformedRate: 0.05,  // 5% malformed rate
  disconnectRate: 0.02, // 2% disconnect rate
});

// Remove all server-level chaos
mock.clearChaos();

Fixture-Level Chaos

Attach a chaos config to individual fixtures so only specific responses experience failures:

chaos-fixture.json json

{
  "fixtures": [
    {
      "match": { "userMessage": "unstable" },
      "response": { "content": "This might fail!" },
      "chaos": {
        "dropRate": 0.3,
        "malformedRate": 0.2,
        "disconnectRate": 0.1
      }
    },
    {
      "match": { "userMessage": "stable" },
      "response": { "content": "This always works." }
    }
  ]
}

Per-Request Headers

Override chaos rates on individual requests using HTTP headers. Values are floats between 0 and 1:

Header	Controls
`x-llmock-chaos-drop`	Drop rate (0–1)
`x-llmock-chaos-malformed`	Malformed rate (0–1)
`x-llmock-chaos-disconnect`	Disconnect rate (0–1)

Per-request chaos via headers ts

// Force 100% disconnect on this specific request
await fetch(`${mock.url}/v1/chat/completions`, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "x-llmock-chaos-disconnect": "1.0",
  },
  body: JSON.stringify({ model: "gpt-4", messages: [...] }),
});

CLI Flags

Set server-level chaos from the command line:

CLI chaos flags bash

npx llmock --fixtures ./fixtures \
  --chaos-drop 0.1 \
  --chaos-malformed 0.05 \
  --chaos-disconnect 0.02

Journal Tracking

When chaos triggers, the journal entry includes a chaosAction field recording which failure mode was applied:

Journal entry with chaos json

{
  "method": "POST",
  "path": "/v1/chat/completions",
  "response": {
    "status": 500,
    "fixture": { "..." },
    "chaosAction": "drop"
  }
}

The chaosAction values are "drop", "malformed", or "disconnect". The status codes are 500 for drop, 200 for malformed, and 0 for disconnect (connection destroyed).

Prometheus Metrics

When metrics are enabled (--metrics), each chaos trigger increments the llmock_chaos_triggered_total counter with an action label:

Metrics output text

# TYPE llmock_chaos_triggered_total counter
llmock_chaos_triggered_total{action="drop"} 3
llmock_chaos_triggered_total{action="malformed"} 1
llmock_chaos_triggered_total{action="disconnect"} 2