System Evolution: Exactly How to Scale Your Infrastructure

Introduction

You shouldn't build an infrastructure designed for one million users when you only have ten.

Over-engineering your scaling patterns early drains your bank account, adds unnecessary complexity, and distracts you from finding product-market fit.

Instead, scale your system in response to real bottleneck metrics.

This guide walks you through exactly what infrastructure you need at each stage of growth, and when to make the jump to the next stage.

The Scaling Journey: 4 Stages

Stage 1: The All-in-One Box (0 to 1,000 Users)

The setup:

┌─────────────────────────────────────────┐
│  Single Virtual Machine (AWS EC2 t3.small) │
├─────────────────────────────────────────┤
│  ├─ Next.js Frontend                   │
│  ├─ Express Backend (Node.js)           │
│  ├─ PostgreSQL Database                 │
│  └─ Redis Cache (optional)              │
└─────────────────────────────────────────┘

Cost: $20-50/month
Deployment: One instance, one deploy script
Monitoring: Minimal (watch server CPU/memory)

What you're optimizing for:

• ✅ Deployment velocity (ship fast)
• ✅ Simplicity (one thing to debug)
• ✅ Cost (as cheap as possible)

When to stay here:

• You have < 1,000 active users
• Your CPU usage is < 50%
• Your database queries complete < 100ms
• One person can handle operations

Example infrastructure:

# Deploy script (simplified)
$ git push main
$ ssh deploy@myapp.com
$ cd myapp && git pull
$ npm install && npm run build
$ pm2 restart app

Monitoring:

Essential metrics:
- CPU usage (alert if > 80%)
- Memory usage (alert if > 90%)
- Disk space (alert if > 80%)
- Response time (alert if p95 > 1 second)
- Error rate (alert if > 1%)

Cost breakdown:

t3.small instance: $20/month
PostgreSQL (on same instance): $0
Domain/DNS: $12/month
Backups: $5/month
Total: ~$37/month

---

Stage 2: Split the Database (1,000 to 50,000 Users)

The trigger:

• Database CPU is consistently > 60%
• Database queries are slowing down
• You're seeing connection pool exhaustion
• Response times are degrading

The setup:

┌──────────────────┐
│  App Server      │
│  (t3.small)      │
├──────────────────┤
│  Next.js API     │
│  Express Backend │
└────────┬─────────┘
         │ TCP Connection
         ▼
┌──────────────────────┐
│  PostgreSQL Database │
│  (Managed RDS)       │
│  db.t3.medium        │
└──────────────────────┘

What changed:

• Database moved to a separate, managed instance
• App can now scale independently of database
• Database can have more resources (CPU, RAM, storage)

Why this helps:

• App can be redeployed without database downtime
• Database performance doesn't affect app performance
• You can scale each independently

When to make this jump:

• Database is the bottleneck (not app CPU)
• You have > 50 database connections
• Queries are taking > 200ms
• You need to backup without app downtime

Cost breakdown:

App instance (t3.small): $20/month
RDS PostgreSQL (db.t3.medium): $100/month
Domain: $12/month
Backups: Automatic in RDS
Total: ~$132/month

How to migrate:

1. Create new RDS instance with same schema
2. Dump data from old database
3. Restore to new RDS
4. Update connection string in app
5. Monitor for issues, rollback if needed

---

Stage 3: Introduce the Cache Layer (50,000 to 200,000 Users)

The trigger:

• Database is hitting resource limits even with a bigger instance
• You're seeing repeated queries for the same data
• p95 latency is creeping up (> 500ms)
• You're hitting database connection limits

The setup:

┌──────────────────┐
│  App Server      │
│  (t3.small)      │
└────┬─────────────┘
     │
     ├─────────────────────┐
     │                     │
     ▼                     ▼
┌─────────────┐    ┌──────────────────┐
│ Redis Cache │    │ PostgreSQL DB    │
│ (ElastiCache)│   │ (db.t3.large)    │
└──────┬──────┘    └──────────────────┘
       │ (Try cache first)
       │ (Fall through to DB if miss)

What Redis caches:

• User sessions (1-2 minute TTL)
• Global configuration (1-hour TTL)
• Frequently accessed data (user profiles, settings)
• Feed rankings (1-hour TTL)
• Rate limiting counters (per-request updates)

Example caching strategy:

async function getUser(userId) {
  // Try cache first
  const cached = await redis.get(`user:${userId}`);
  if (cached) {
    return JSON.parse(cached);
  }
  
  // Cache miss, hit database
  const user = await db.users.findById(userId);
  
  // Store in cache for 5 minutes
  await redis.set(`user:${userId}`, JSON.stringify(user), 'EX', 300);
  
  return user;
}

When to introduce cache:

• You're querying the same data repeatedly
• You have slowly-changing data (configs, user profiles)
• Database CPU is high even with bigger instance
• You want faster response times

Cost breakdown:

App instance: $20/month
Redis (cache.t3.small): $30/month
RDS PostgreSQL (db.t3.large): $200/month
Domain: $12/month
Total: ~$262/month

Important notes:

• Cache = complexity (cache invalidation is hard)
• Only cache data you can afford to be stale
• Only cache frequently accessed data
• Cache doesn't help with write-heavy operations

---

Stage 4: Horizontally Scale (200,000+ Users)

The trigger:

• Single app server CPU is consistently > 80%
• You can't upgrade instance size further (hitting AWS limits)
• You need to handle traffic spikes
• You need redundancy (app can't be single point of failure)

The setup:

┌──────────────────────────────────────┐
│  Load Balancer (AWS ALB)              │
└─────────────────┬──────────────────┬─┘
                  │                  │
        ┌─────────▼────────┐  ┌──────▼──────────┐
        │ App Server #1    │  │ App Server #2   │
        │ (t3.small)       │  │ (t3.small)      │
        │ Auto-scaled      │  │ Auto-scaled     │
        └─────────┬────────┘  └──────┬──────────┘
                  │                  │
                  └──────────┬───────┘
                             │
                    ┌────────▼────────┐
                    │ Redis Cache     │
                    │ (ElastiCache)   │
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
                    │ PostgreSQL DB   │
                    │ (db.t3.xlarge)  │
                    └─────────────────┘

How it works:

1. Load balancer receives all traffic
2. Forwards to healthy app instances
3. If one instance fails, traffic routes to others
4. If traffic spikes, auto-scaling adds new instances
5. All instances share same database and cache

Auto-scaling config:

Min instances: 2
Max instances: 10
Target CPU: 70%
Scale up: When avg CPU > 70% for 2 minutes
Scale down: When avg CPU < 40% for 5 minutes

Deployment with auto-scaling:

# Old way (single server)
$ ssh deploy@app.com
$ git pull && npm run build && npm restart
# ❌ Downtime during deployment

# New way (auto-scaled)
# 1. Build new Docker image
$ docker build -t myapp:v2.0 .
$ docker push myapp:v2.0

# 2. Update auto-scaling group config
$ aws autoscaling update-launch-configuration \
  --launch-configuration-name myapp \
  --image-id ami-newversion

# 3. Gradually replace instances (rolling deployment)
$ aws autoscaling start-instance-refresh \
  --auto-scaling-group-name myapp
# ✓ Zero downtime, instances gradually replaced

Cost breakdown:

Load Balancer: $16/month
App instances (2-10): $20-100/month (auto-scaled)
Redis (cache.t3.medium): $60/month
RDS PostgreSQL (db.t3.xlarge): $500+/month
Domain: $12/month
Total: $600-700+/month (scales with usage)

---

Advanced Stages (Beyond 1M Users)

If you get here, you've "made it." Additional stages:

Stage 4.5: Database Read Replicas

Primary DB (writes)
  ├─ Read Replica 1 (analytics queries)
  ├─ Read Replica 2 (API reads)
  └─ Read Replica 3 (reporting)

Stage 5: Separate Microservices

API Layer (search, users)
Billing Service (transactions, invoices)
Analytics Service (data warehouse, reports)
Background Jobs Service (emails, exports)

Stage 6: Global Distribution

CDN (CloudFront, Cloudflare) for static assets
Regional databases for low latency
Service mesh for inter-service communication

The Engineering Principle

Scale vertically (bigger servers) as long as it's cost-effective.

Scale horizontally (more servers) only when vertical hits a hard ceiling.

Cost vs. Resources

Vertical scaling (get bigger servers):
$0 ─────────────────────────┐
    └─ t3.small → t3.medium → t3.large → t3.xlarge
    
Cost is linear, operations are simple

Horizontal scaling (get more servers):
$0 ────────────┐─────────────────────
    └─ 1 server → 2 → 4 → 8 → 16 → 32
    
Cost is linear, but operations become complex

When to switch:

• Vertical: When a bigger instance is 30% cheaper than running 2 smaller ones
• Horizontal: When vertical scaling hits limits (DB size, instance size limits) or you need redundancy

Real-World Example Timeline

Month 1: 100 users
┗ Single t3.small instance ($20/month)
  All-in-one box works great

Month 6: 5,000 users
┗ Still single t3.small, but database is getting slow
  Need to split database

Month 12: 50,000 users
┗ App: t3.small ($20/month)
  Database: RDS db.t3.medium ($100/month)
  Cache: ElastiCache Redis ($30/month)
  Total: ~$150/month

Month 18: 200,000 users
┗ Load balancer + 2-4 app servers (auto-scale)
  Database: RDS db.t3.xlarge ($500/month)
  Cache: Redis cluster
  Total: ~$700/month

Month 24: 1,000,000 users
┗ Multiple regions
  Microservices
  Database replicas
  CDN for static assets
  Total: $5,000+/month

Monitoring at Each Stage

Stage	Key Metrics	Alerts
Stage 1	CPU, Memory, Disk	CPU > 80%, Disk > 80%
Stage 2	DB latency, connections	P95 > 200ms, connections > 80%
Stage 3	Cache hit rate	Hit rate < 60%
Stage 4	Instance health, traffic distribution	Instance down, CPU > 80%

Key Takeaways

1. Start simple — One box is fine for 0-1,000 users
2. Scale in response to bottlenecks — Don't over-engineer early
3. Measure before scaling — Know what's actually slow
4. Vertical before horizontal — Bigger servers are simpler
5. Use managed services — RDS, ElastiCache, Load Balancers handle complexity
6. Monitor constantly — Know when you're approaching limits
7. Plan migrations — Know the next step before you need it

The best infrastructure is the one you don't have to build because you're scaling the one you have.

---

Remember: Stripe didn't build a global infrastructure on day one. They started on Heroku. As they grew, they migrated to AWS infrastructure. You don't need to skip steps just because you know where you're headed.

Scale your infrastructure to match your business growth, not your imagination about future scale.