Chaos engineering: finding weaknesses in production
Chaos engineering is the practice of deliberately injecting failure into production (or production-like) systems to discover weaknesses before they cause incidents. Tools like Chaos Monkey, Gremlin, and AWS Fault Injection Service terminate EC2 instances, partition networks, introduce latency at the infrastructure level, and saturate CPU — while the system is serving real traffic.
The philosophy: systems will fail in unexpected ways. Better to discover those failure modes deliberately and safely (with runbooks and a team watching) than during an actual incident at 2am. Chaos engineering is inherently production-focused, requires organizational buy-in, and carries real risk.
slowdep: shift-left chaos for developers
slowdep operates at the test level, not the production level. Instead of injecting failures into a running system, it wraps functions in tests to simulate the kind of latency and error conditions that chaos engineering would find in production.
Think of it as "shift-left chaos" — finding the bugs in dev and CI that chaos engineering would find in production. The bugs that circuit breakers miss, retries don't handle, and timeouts don't catch properly are bugs that can be found during development — if your tests have realistic latency.
"Shift-left" means moving testing earlier in the development cycle (left on the timeline). slowdep shifts chaos testing from production all the way back to the developer's laptop and CI.
Chaos engineering vs slowdep
| dimension | chaos engineering | slowdep |
|---|---|---|
| environment | production or staging | development and CI tests |
| risk level | high — real users may be affected | zero — runs in test process only |
| cost | high — tooling, team time, incident risk | zero — free npm package |
| team required | SRE team, runbooks, monitoring | individual developer |
| feedback loop | hours — deploy, run experiment, analyze | seconds — test run |
| what it tests | system-level resilience (infra, networking, dependencies) | code-level resilience (retry logic, timeouts, circuit breakers) |
| when to use | after code is in production and validated | during development and every CI build |
| granularity | infrastructure-level (hosts, networks, zones) | function-level (per dependency method) |
Complementary goals, different phases
slowdep doesn't replace chaos engineering — it complements it by addressing the earlier phase of the development cycle. A typical resilience testing strategy uses both:
SLOWDEP_ENABLED=true)
to prevent latency regressions from being merged. A circuit breaker misconfiguration
won't make it to production if it fails a CI test that simulates the trip condition.
The same bug, found earlier
Consider a circuit breaker misconfiguration: the threshold is set to 10 failures, but in production a 30% error rate at 3s p99 trips the breaker with only 4 failures (because the failures come so slowly that the sliding window sees them as dense).
Chaos engineering would find this during a production game day, with real users affected. slowdep finds it during a test run, with a reproducible, fixable scenario.
// This test would catch the circuit breaker misconfiguration // before chaos engineering would need to find it in production import { withLatency } from 'slowdep'; it('circuit breaker trips at correct failure count under realistic latency', async () => { const degraded = withLatency( async () => { throw new Error('downstream error'); }, { p50: 500, p99: 3000, errorRate: 1.0 } // realistic: slow AND failing, like a degraded production service ); const cb = new CircuitBreaker({ threshold: 10, window: 5000 }); let failFastCount = 0; for (let i = 0; i < 15; i++) { const start = Date.now(); await cb.execute(() => degraded()).catch(() => { if (Date.now() - start < 10) failFastCount++; // <10ms = fail-fast }); } // after 10 failures, subsequent calls should fail-fast // if this fails: your threshold is misconfigured for realistic latency expect(failFastCount).toBeGreaterThan(0); });