Introduction
If you've been building software lately, you've felt the frustration:
Your AI code editor has a rough morning. GitHub Actions hangs for an hour. Stripe is experiencing "unexpected downstream turbulence." Your cloud hosting provider's DNS goes down for 20 minutes. A seemingly small outage at one service spirals into cascading failures across your entire stack.
It feels like every single week something is breaking.
It's easy to shake your head and assume engineers are just shipping sloppy code. "Why can't these big companies keep their infrastructure stable?" you think.
But the reality runs much deeper. Software isn't getting worse because engineers became lazy. It's getting less stable because our systems have never been this hyper-connected.
This guide explains why modern software is inherently more fragile—and what you can do about it.
The Era of Hyper-Connected Systems
The Old Way: Monolithic, Isolated Apps
┌─────────────────────────────────┐
│ Your Entire Application │
│ ├─ Authentication │
│ ├─ Billing │
│ ├─ Analytics │
│ └─ API Logic │
│ │
│ Single point of failure │
│ but failure is predictable │
└─────────────────────────────────┘
Runs on: Your server, in your data center
Depends on: Exactly nothing (maybe a database)
If something broke, it broke locally and predictably. You could see the logs, understand the issue, and fix it.
The Modern Way: Distributed, Hyper-Connected Stacks
Your App (Next.js)
↓ (depends on) ↓
┌─────────────────────────────────────────────────────────┐
│ │
│ AI IDE Integration (Claude Code) │
│ ↓ │
│ GitHub Actions (CI/CD) │
│ ↓ │
│ Third-party Auth (Auth0, Clerk) │
│ ↓ │
│ Payment Processing (Stripe) │
│ ↓ │
│ Managed Database (Supabase, RDS) │
│ ↓ │
│ Cloud Storage (S3) │
│ ↓ │
│ Analytics (PostHog, Segment) │
│ ↓ │
│ Email Delivery (SendGrid) │
│ ↓ │
│ Monitoring (Sentry, DataDog) │
│ │
│ A single network blip down the chain │
│ breaks the whole pipeline │
└─────────────────────────────────────────────────────────┘
When you look at a modern stack, your development environment relies on live AI code generation streams. Your authentication relies on a third-party managed provider. Your serverless database handles scaling behind a managed gateway. Your payments go through a payment processor. Your emails are sent through a third-party service.
We have traded isolation for raw power and delivery speed.
Each dependency you add buys you velocity (you don't have to build auth, you use Auth0). But it also buys you fragility (if Auth0 is down, your users can't log in).
Why Modern Systems Are Inherently More Fragile
Reason #1: Dependencies Compound Risk
Each external dependency introduces risk:
1 dependency = 99.9% uptime = 43 minutes downtime/month
10 dependencies = 99.9%^10 = 99% uptime = 7.2 hours downtime/month
50 dependencies = 99.9%^50 = 95% uptime = 36 hours downtime/month
If each service has 99.9% uptime and you depend on 50 of them, your system is down 1.5 days per month on average.
This isn't a failure of engineering. It's a mathematical inevitability.
Reason #2: Cascading Failures
One service failing doesn't just affect that service. It cascades:
Scenario: Stripe is slow (taking 30 seconds to respond)
Your code:
const charge = await stripe.charges.create(...) // Times out after 30s
await database.invoices.create(...) // Never executes
Result:
- User is charged
- Invoice is NOT created
- System is in an inconsistent state
- Support tickets flood in
- You page the on-call engineer at 2 AM
Reason #3: Network Unreliability
The fundamental problem is that networks are unreliable:
The Network is Unreliable:
├─ Packets get lost
├─ Requests timeout
├─ Connections drop mid-transfer
├─ DNS resolution fails
├─ Firewalls block traffic
└─ Cloud provider regions go down
Your Code Must Handle All of These
You can't prevent network failures. You can only handle them gracefully.
How to Build for an Unreliable Internet
If you want your platform to stay up when the dependencies around you are shaking, you have to build with defensiveness in mind.
Defensive Pattern #1: Graceful Degradation
If a non-essential service fails, your main application should continue working.
Bad approach:
// If analytics fails, the entire request fails
async function handleRequest(req, res) {
const user = await getUser(req.userId);
// This single call can break the entire request
await analytics.track('user_action', { userId: user.id });
res.json({ success: true });
}
Good approach:
// If analytics fails, the user still gets a response
async function handleRequest(req, res) {
const user = await getUser(req.userId);
// Wrap non-essential calls in try-catch
try {
await analytics.track('user_action', { userId: user.id });
} catch (error) {
// Log the error but don't fail the request
console.error('Analytics failed:', error);
// User still gets a response
}
res.json({ success: true });
}
Classify your dependencies:
CRITICAL:
- Authentication
- Payment processing
- Core database
NON-CRITICAL:
- Analytics
- Email notifications
- Tracking pixels
- A/B testing
Critical paths should fail fast and visibly. Non-critical paths should fail silently.
Defensive Pattern #2: Smart Retry Policies
Don't let your app immediately throw a 500 error if a request fails on the first try.
Bad approach:
// One failure = immediate error
const response = await fetch(externalService);
if (!response.ok) {
throw new Error('Service failed');
}
Good approach: Exponential Backoff with Jitter
async function fetchWithRetries(url, maxRetries = 3) {
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
const response = await fetch(url, { timeout: 5000 });
if (response.ok) return response;
} catch (error) {
// Calculate exponential backoff with jitter
const delay = Math.min(
1000 * Math.pow(2, attempt) + Math.random() * 1000,
10000 // Max 10 second backoff
);
if (attempt < maxRetries - 1) {
await sleep(delay);
continue;
}
}
}
throw new Error('Max retries exceeded');
}
Why exponential backoff with jitter?
- Exponential backoff: First retry at 2s, then 4s, then 8s (don't hammer failing service)
- Jitter: Add randomness (100ms ± 50ms) to prevent the thundering herd (all clients retrying at exactly the same time)
This prevents you from accidentally DDOSing an external service when it's trying to recover.
Defensive Pattern #3: Circuit Breakers
Stop calling a failing service immediately. Wait before retrying.
class CircuitBreaker {
constructor(service, threshold = 5, timeout = 60000) {
this.service = service;
this.failureCount = 0;
this.threshold = threshold; // Fail after 5 failures
this.timeout = timeout; // Wait 60 seconds before retry
this.state = 'CLOSED'; // CLOSED (working), OPEN (failing), HALF_OPEN (testing)
this.lastFailureTime = null;
}
async call(fn) {
// If circuit is open, check if we should try again
if (this.state === 'OPEN') {
if (Date.now() - this.lastFailureTime > this.timeout) {
this.state = 'HALF_OPEN'; // Try one request
} else {
throw new Error('Circuit breaker is OPEN');
}
}
try {
const result = await fn();
// Success! Reset the circuit
this.failureCount = 0;
this.state = 'CLOSED';
return result;
} catch (error) {
this.failureCount++;
this.lastFailureTime = Date.now();
if (this.failureCount >= this.threshold) {
this.state = 'OPEN'; // Stop calling this service
}
throw error;
}
}
}
// Usage
const stripeCircuit = new CircuitBreaker(stripe);
try {
await stripeCircuit.call(() => stripe.charges.create(...));
} catch (error) {
if (error.message.includes('Circuit breaker')) {
// Stripe is down, use a fallback
return handlePaymentFallback();
}
throw error;
}
Defensive Pattern #4: Timeouts Everywhere
Don't let requests hang forever. Set explicit timeouts:
// Bad: No timeout (request could hang forever)
const response = await fetch(url);
// Good: Explicit timeout
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 5000);
try {
const response = await fetch(url, { signal: controller.signal });
clearTimeout(timeout);
return response;
} catch (error) {
if (error.name === 'AbortError') {
throw new Error('Request timeout');
}
throw error;
}
Defensive Pattern #5: Bulkheads / Isolation
Isolate different parts of your system so one failure doesn't cascade.
// Without bulkheads:
async function handleRequest(req, res) {
const user = await getUser(); // Shared connection pool
const orders = await getOrders(); // Same pool
const recommendations = await getAI(); // Same pool
// If getAI is slow, it consumes all connections
// getUser and getOrders start timing out too
}
// With bulkheads:
const userPool = createPool(maxConnections: 10);
const orderPool = createPool(maxConnections: 10);
const aiPool = createPool(maxConnections: 5);
async function handleRequest(req, res) {
const user = await userPool.query(getUser); // Own pool
const orders = await orderPool.query(getOrders); // Own pool
const recommendations = await aiPool.query(getAI); // Own pool, smaller
// If getAI is slow, only AI requests are affected
// User and order requests proceed normally
}
Key Takeaways
- Modern systems are inherently more fragile — More dependencies = more risk
- Cascading failures are inevitable — One service failing cascades through the stack
- Build for defensiveness — Graceful degradation, retries, circuit breakers
- Classify dependencies — Critical vs non-critical, fail differently
- Timeout everything — Don't let requests hang forever
- Use exponential backoff — With jitter, don't hammer failing services
- Isolate components — Use bulkheads to prevent cascade failures
Your goal isn't to prevent failures (you can't). Your goal is to gracefully handle them when they happen.
Remember: The more dependencies you have, the more defensive your code needs to be. Every external API you call is a potential source of failure. Plan for it.
Ready to start building?
Explore the most comprehensive directory of APIs for Nigerian developers and find exactly what you need.
Browse the API Directory
