slowdep/Guides/AI applications

Testing AI applications

LLM API calls are 600ms–8s with high variance. slowdep's OpenAI and Anthropic presets let you test real latency behavior without burning tokens.

The problem

The AI latency problem

LLM API calls are unlike any other dependency. They're slow (600ms–8s for typical responses), highly variable (prompt length, output length, and server load all affect latency), and the tail can be extreme — a long generation on a loaded server can take 30+ seconds. Streaming adds another dimension: the latency before the first token arrives is different from the total latency.

Testing this with setTimeout(r, 1000) is meaningless. Your "thinking..." indicator that renders for exactly 1000ms in tests might flash for 200ms or hang for 12 seconds in production. Rate limits kick in under real load. Cold starts add 2–5 seconds. The only way to test this correctly is with realistic latency distribution — without burning API tokens.

Cost savings

Every test run that calls a real LLM costs money. With slowdep, wrap your AI client with a mock that returns a fake response, add realistic latency, and test all latency-dependent behavior for free.

OpenAI preset

The openai preset

The 'openai' preset is configured with p50: 800ms, p99: 8s — based on typical GPT-4 latency for medium-length completions. Most calls complete in under a second. 1% exceed 8 seconds. 

import { withLatencyAll } from 'slowdep';
import OpenAI from 'openai';

// in your test setup
const mockOpenAI = {
  chat: {
    completions: {
      create: async (_opts: any) => ({
        choices: [{ message: { content: 'Mock response text' } }],
        usage: { prompt_tokens: 100, completion_tokens: 50 },
      }),
    },
  },
};

// wrap with realistic OpenAI latency
const slowOpenAI = withLatencyAll(mockOpenAI.chat.completions, 'openai');
// p50: 800ms, p99: 8000ms — lognormal distribution

const response = await slowOpenAI.create({ model: 'gpt-4', messages: [...] });
// behaves like a real OpenAI call, latency-wise
Anthropic preset

The anthropic preset

The 'anthropic' preset uses p50: 600ms, p99: 7s — Claude models tend to be slightly faster at median but have a similar tail. Use this when wrapping Anthropic SDK calls. 

import { withLatency } from 'slowdep';
import Anthropic from '@anthropic-ai/sdk';

// mock message creator with realistic response
const mockCreate = async (_opts: Anthropic.MessageCreateParams) =>
  (await import('./fixtures/anthropic-response.json')) as Anthropic.Message;

// wrap with the anthropic preset
const slowCreate = withLatency(mockCreate, 'anthropic');
// p50: 600ms, p99: 7000ms

jest.spyOn(anthropic.messages, 'create').mockImplementation(slowCreate);

// now all tests using anthropic.messages.create get realistic latency
Streaming

Testing streaming UX

slowdep wraps the function call itself — the delay is applied before the async function resolves. For streaming LLM responses, this models the "time to first token" latency: the delay before the stream starts flowing. Test that your "thinking..." indicator renders correctly during this initial wait period. 

it('shows thinking indicator during time-to-first-token', async () => {
  jest.useFakeTimers();

  // mock streaming response: slowdep delays before stream starts
  const mockStream = withLatency(
    async () => createMockStream(['Hello', ' world', '!']),
    'anthropic'
  );
  jest.spyOn(anthropic.messages, 'stream').mockImplementation(mockStream);

  render(<ChatMessage prompt="Hello" />);

  // before stream starts, thinking indicator should be visible
  expect(screen.getByTestId('thinking-indicator')).toBeInTheDocument();
  expect(screen.queryByTestId('message-content')).not.toBeInTheDocument();

  // advance past p50 — stream starts flowing
  jest.advanceTimersByTime(800);
  await waitFor(() => {
    expect(screen.getByTestId('message-content')).toBeInTheDocument();
  });

  jest.useRealTimers();
});
Rate limits

Testing retry on rate limit

AI APIs rate-limit heavily, especially during peak hours. A 429 response is a transient error that should trigger exponential backoff. Set errorRate: 0.1 to simulate occasional rate limit errors, combined with high latency to test realistic retry behavior. 

const rateLimitedOpenAI = withLatency(mockCreate, {
  p50: 800,
  p99: 8000,
  errorRate: 0.1,  // 10% rate limit errors
});

it('retries on rate limit with exponential backoff', async () => {
  let callCount = 0;
  const tracked = withLatency(
    async (...args) => { callCount++; return mockCreate(...args); },
    { p50: 800, p99: 8000, errorRate: 0.5 } // high rate for reliable test
  );

  jest.spyOn(global, 'setTimeout').mockImplementation((fn) =>
    globalThis.setTimeout(fn, 0)
  );

  const results = await Promise.allSettled(
    Array.from({ length: 10 }, () => callWithRetry(tracked, 3))
  );
  // some calls succeeded via retry
  expect(callCount).toBeGreaterThan(10);
  jest.restoreAllMocks();
});
Token savings

Token cost savings in CI

The pattern for zero-cost AI testing: return a fixture response from the mock function, then wrap it with slowdep to add realistic latency. Your tests exercise all latency-dependent code paths — timeout logic, streaming UX, retry behavior, loading states — without a single real API call. 

// test/setup.ts — wraps AI clients globally in test environment
import { withLatencyAll } from 'slowdep';

if (process.env.NODE_ENV === 'test') {
  // replace real client with a mock that has realistic latency
  jest.mock('openai', () => ({
    default: class MockOpenAI {
      chat = {
        completions: withLatencyAll({
          create: async () => FIXTURE_RESPONSE,
        }, 'openai'),
      };
    },
  }));

  jest.mock('@anthropic-ai/sdk', () => ({
    default: class MockAnthropic {
      messages = withLatencyAll({
        create: async () => ANTHROPIC_FIXTURE,
      }, 'anthropic');
    },
  }));
}
Agent chains

Multi-model agent testing

When building LLM agents that chain multiple model calls, each call accumulates latency. Wrap each model with its own preset so you get realistic end-to-end agent latency in tests. 

import { withLatency } from 'slowdep';

// each step in the agent chain gets its own realistic latency
const plannerCall  = withLatency(mockPlannerLLM,  'openai');     // p50: 800ms
const toolCallLLM  = withLatency(mockToolLLM,    'anthropic');  // p50: 600ms
const summaryCall  = withLatency(mockSummaryLLM, 'openai');     // p50: 800ms
const dbLookup     = withLatency(mockDb.query,   'postgres');  // p50: 5ms

it('completes 3-step agent within 5 second timeout', async () => {
  const result = await Promise.race([
    runAgent({ planner: plannerCall, tool: toolCallLLM, summary: summaryCall, db: dbLookup }),
    new Promise((_, r) => setTimeout(() => r(new Error('agent timeout')), 15000)),
  ]);
  // p50 chain: 800 + 600 + 800 + 5 ˜ 2.2s — well within 15s timeout
  // p99 chain: 8000 + 7000 + 8000 + 200 ˜ 23s — might timeout, as expected
  expect(result).toHaveProperty('answer');
});