Cascading failures
When a downstream service becomes slow, requests to your service accumulate while waiting for it. Your thread pool, connection pool, or event loop queue fills up. Incoming requests start queuing behind stuck ones. Your service becomes slow too — and now your upstream is waiting on you. The failure cascades up the call chain.
A circuit breaker short-circuits this: once a dependency is detected as unhealthy, subsequent calls fail immediately (fast-fail) instead of adding to the queue. Your resources are freed to handle requests that can actually succeed.
The three states
Why circuit breakers can't be tested with mocks alone
A circuit breaker that trips on latency — not just on errors — needs real latency to trigger. If you mock your downstream to throw instantly, you test error-rate tripping but not latency-based tripping. The most dangerous production scenario (slow service, not failed service) goes untested.
import { withLatency } from 'slowdep'; // Simulates a service that's slow AND error-prone — the trip condition const degradedDb = withLatency(db.query, { p50: 800, // 800ms median — well above typical 5ms p99: 5000, // 5s worst case errorRate: 0.3, // 30% of calls fail });
A complete circuit breaker test
import { withLatency } from 'slowdep'; import { CircuitBreaker } from 'opossum'; // or cockatiel, etc. test('circuit opens after repeated failures', async () => { // 1. Set up a reliably failing dependency const failingDb = withLatency(db.query, { p50: 100, p99: 500, errorRate: 1.0, }); const breaker = new CircuitBreaker(failingDb, { errorThresholdPercentage: 50, volumeThreshold: 5, }); // 2. Trip the circuit — run enough calls to cross the threshold for (let i = 0; i < 6; i++) { await breaker.fire().catch(() => {}); } // 3. Circuit should now be open — next call fails immediately await expect(breaker.fire()).rejects.toMatchObject({ message: expect.stringContaining('open'), }); expect(breaker.opened).toBe(true); }); test('circuit closes after successful probe', async () => { // Start with open circuit, then simulate recovery breaker.open(); // force open // Swap in a healthy dependency const healthyDb = withLatency(db.query, { p50: 5, p99: 50, errorRate: 0, }); // After cooldown, half-open probe should succeed and close circuit jest.advanceTimersByTime(30000); // skip cooldown await breaker.fire(); expect(breaker.closed).toBe(true); });
Key threshold parameters
| parameter | typical value | description |
|---|---|---|
| errorThreshold | 50% | Failure rate above which the circuit opens |
| volumeThreshold | 5–20 req | Minimum request count before the threshold is evaluated — prevents tripping on the first error |
| windowSize | 10s | Rolling time window for counting failures |
| cooldown | 30s | Time the circuit stays open before transitioning to half-open |
| latencyThreshold | p99 of dependency | Requests slower than this count as failures, even if they eventually succeed |