System Evolution: How to Scale Your Infrastructure

Introduction

You shouldn't build an infrastructure designed for one million users when you only have ten.

Over-engineering your scaling patterns early drains your bank account and distracts you from finding product-market fit.

Instead, scale your system in response to real bottleneck metrics.

This guide walks you through exactly what infrastructure you need at each stage of growth.

Stage 1: The All-in-One Box (0 to 1,000 Users)

The Setup

Single virtual machine with everything:

• Next.js frontend
• Express backend
• PostgreSQL database
• Redis cache (optional)

Cost: $20-50/month Deployment: One instance, one deploy script

When to stay here:

• You have less than 1,000 active users
• Your CPU usage is below 50%
• Database queries complete in under 100ms

---

Stage 2: Split the Database (1,000 to 50,000 Users)

The Trigger

• Database CPU consistently above 60%
• Database queries slowing down
• Connection pool exhaustion
• Response times degrading

What Changed

• Database moves to separate, managed instance
• App can scale independently
• Database has more resources

Why this helps:

• App can be redeployed without database downtime
• Database performance doesn't affect app performance
• You can scale each independently

---

Stage 3: Introduce Cache Layer (50,000 to 200,000 Users)

The Trigger

• Database hitting resource limits even with bigger instance
• Repeated queries for the same data
• p95 latency creeping up above 500ms

What Redis Caches

• User sessions (1-2 minute TTL)
• Global configuration (1-hour TTL)
• Frequently accessed data
• Rate limiting counters

When to introduce cache:

• Querying same data repeatedly
• Have slowly-changing data
• Database CPU still high even with bigger instance

---

Stage 4: Horizontally Scale (200,000+ Users)

The Trigger

• Single app server CPU consistently above 80%
• Can't upgrade instance size further
• Need redundancy

The Setup

• Load balancer (AWS ALB)
• Multiple app servers (2-10, auto-scaled)
• Shared database and cache
• Zero-downtime deployments

Auto-Scaling Config

Min instances: 2 Max instances: 10 Target CPU: 70%

---

The Engineering Principle

Scale vertically (bigger servers) as long as it's cost-effective.

Scale horizontally (more servers) only when vertical hits a hard ceiling.

When a bigger instance is 30% cheaper than running 2 smaller ones, go vertical. When you hit size limits or need redundancy, go horizontal.

Real-World Example Timeline

Month 1: 100 users

• Single t3.small instance (all-in-one)

Month 6: 5,000 users

• Still single t3.small, but database getting slow

Month 12: 50,000 users

• App: t3.small
• Database: RDS db.t3.medium
• Cache: Redis
• Total: ~$150/month

Month 18: 200,000 users

• Load balancer + 2-4 app servers (auto-scaled)
• Database: RDS db.t3.xlarge
• Total: ~$700/month

Related Guides

Before scaling, ensure your architecture is appropriate. See Microservices vs. Monoliths to understand when to split services.

Also see Platform Stability: Is Software Less Reliable Now? to understand how to build defensively as you add dependencies.

For development practices that support scaling, see Stop Over-Engineering Your Local Environment.

And ensure you have proper deployment practices with The Only Pre-Deployment Checklist You Need.

Key Takeaways

1. Start simple - One box is fine for 0-1,000 users
2. Scale in response to bottlenecks - Don't over-engineer early
3. Measure before scaling - Know what's actually slow
4. Vertical before horizontal - Bigger servers are simpler
5. Use managed services - They handle complexity
6. Monitor constantly - Know when approaching limits
7. Plan migrations - Know the next step before you need it

For deeper understanding, see the AWS Scalability Best Practices and Heroku's Scaling Patterns.

---

Remember: Stripe didn't build global infrastructure on day one. They started on Heroku. As they grew, they migrated to AWS infrastructure. Scale your infrastructure to match your business growth, not your imagination.