# How to scale backends

The backend is the compute tier between your frontend and your data, and scaling it means handling more requests per second without tipping over. The good news is that a backend done right scales almost as easily as static files - you add more identical servers. The whole art is in *earning* that word "identical," then spreading load across the copies, taking work off the request path, and staying up when a piece fails. This post explains how each of those actually works, building on the [core building blocks](/concepts-you-should-know-about-system-designing-part-1).

## Vertical first, because it is free

The simplest scaling is **vertical**: give the one server more CPU and RAM. It buys real time with zero code change, and you should do it early. But it has two hard limits. There is a physical ceiling - you cannot buy an infinitely large machine, and the price per unit of performance climbs steeply near the top. And a single big server is a single point of failure: when it reboots or dies, everything is down. So vertical scaling is a runway, not a destination. **Horizontal scaling** - many servers sharing the load - is the real answer, and everything below exists to make it work.

## Statelessness is the entire enabler

Horizontal scaling only works if *any* instance can serve *any* request. The blocker is **state** kept in a server's own memory - a logged-in user's session, an in-process cache, a half-finished upload, an open WebSocket. If instance A holds your session in its RAM and your next request lands on instance B, you are suddenly logged out. So the rule is: **keep the service tier stateless** and push state outward.

Two common ways to externalize session state, with a real trade-off:

- **Shared session store.** Put the session in a store every instance can reach, like [Redis](https://redis.io), keyed by a session id in the client's cookie. Any instance looks it up. You can revoke a session instantly (delete the key), but every request pays a lookup.
- **Signed token (JWT).** Put the session data in a token the client carries, signed so the server can trust it without a lookup. Truly stateless and fast, but you cannot easily revoke one before it expires, and the token travels on every request.

Whichever you pick, the corollary is the same: **no sticky sessions.** Sticky sessions pin a user to one instance, which defeats even load spreading and turns that instance's death into lost sessions. Make every instance an identical, disposable clone from the same container image, and you can add, kill, and replace them at will.

## Spreading load, then automating it

With stateless instances, a **load balancer** sits in front and distributes requests. It does two jobs at once. It spreads traffic - by round-robin, by least-connections (send the next request to the instance handling the fewest, which matters when request durations vary), or by hashing a key. And it **health-checks** every instance and stops routing to any that fail, so a dead instance drops out within seconds instead of black-holing requests. An L4 balancer routes on IP/port (fast, content-blind); an L7 balancer reads HTTP and can route on path or headers and terminate TLS. Together, "many instances + a load balancer" is what converts raw servers into both throughput and fault tolerance.

Then make capacity elastic. An **autoscaler** watches a metric - CPU, request rate, or queue depth - and adds instances when it crosses a threshold and removes them when load falls. On [Kubernetes](https://kubernetes.io/) this is the Horizontal Pod Autoscaler; on cloud VMs, an auto-scaling group. Two things to respect: new instances take time to boot and warm up, so scaling reacts with a lag - which is why you scale out at, say, 60% CPU, not 95%; and you set a cooldown so the system does not thrash, adding and removing instances every few seconds around the threshold.

## Take work off the request path

A huge amount of perceived slowness is doing work synchronously that the user never needed to wait for. When someone uploads a video, the response should return the instant it is saved; transcoding it, generating thumbnails, notifying followers, and sending email should all happen *after*. The tool is a **message queue** ([Kafka](https://kafka.apache.org/), [RabbitMQ](https://www.rabbitmq.com/), SQS). The request enqueues a job and returns immediately; a separate pool of **worker** processes drains the queue at its own pace. This does three distinct things:

- **Cuts request latency** - the user waits only for the fast part.
- **Absorbs spikes** - a burst that would overwhelm synchronous processing just makes the queue longer, and the workers drain it steadily instead of the system collapsing. The queue is a shock absorber.
- **Decouples services** - the uploader does not call the transcoder directly, so they scale, deploy, and fail independently.

Two mechanics you must get right. Queues deliver **at-least-once**, meaning a job can arrive twice (a worker crashes after doing the work but before acknowledging), so workers must be **idempotent** - processing the same job twice has the same effect as once, usually enforced with a job id you record as done. And a job that keeps failing should land in a **dead-letter queue** after N attempts, so one poison message does not block the pipeline forever. You scale the workers by watching queue depth: lag growing means add workers.

## Cache before you compute

The fastest request is the one you never fully process. A cache in front of expensive reads and repeated computation means most requests are served from memory instead of hitting the database. *Which* caching pattern (cache-aside, read-through, write-through) and *what to evict when it fills* are decisions in their own right, covered in the caching posts - but the headline for scaling is blunt: a well-placed cache often removes more load than another ten servers would.

## Split the monolith only when it pays

When one service does too much and its parts scale differently, you can break it into **microservices** that scale independently - the video-processing service gets more workers without touching auth - behind an **API gateway** that centralizes routing, authentication, and rate limiting. This is powerful and genuinely expensive: every in-process call becomes a network hop that can fail, a single user action can span services (so you need distributed tracing to debug it and sagas or careful design to keep data consistent), and you now operate many deployables instead of one. So split on a real boundary - a part with a different scaling profile, or a different team owning it - not on day one. A well-built monolith scales horizontally for a long time. On the read side specifically, separating the read and write paths with [CQRS](/command-query-responsibility-segregation-cqrs) lets reads scale independently of writes.

## Staying up under load

Scaling is not only throughput; it is not falling over when something breaks. Four patterns are non-negotiable at scale, and each has a concrete mechanism:

- **Rate limiting** caps how much any client can send. The common algorithm is a **token bucket**: a bucket refills at a fixed rate up to a cap, each request spends a token, and an empty bucket means the request is throttled. It allows short bursts (spend the accumulated tokens) while bounding the sustained rate, and it protects you from both abuse and an accidental client loop.
- **Timeouts and retries.** Every outbound call needs a timeout, or one slow dependency hangs every request thread until the whole service stalls. Retries recover from transient blips, but naive retries cause a **thundering herd** - a thousand clients retrying in lockstep hammer the recovering service back down. So retry with **exponential backoff and jitter** (randomized delays), and only for idempotent operations.
- **Circuit breaker.** Wrap calls to a flaky dependency in a breaker with three states: **closed** (calls flow), **open** (after too many failures, calls fail fast without even trying, giving the dependency room to recover), and **half-open** (after a cooldown, let a trickle through to test if it is healthy, then close or re-open). This stops you from piling requests onto something already on fire.
- **Bulkheads and backpressure.** Isolate resource pools (separate thread/connection pools per dependency) so one slow downstream cannot exhaust the resources the rest of the system needs - the bulkhead. And when consumers cannot keep up, signal producers to slow down (backpressure) or shed load deliberately, rather than accepting work until you crash.

Without these, scaling up just means failing bigger and more expensively.

## The full architecture

Put the pieces together and one picture describes a scalable backend - a load balancer over a stateless, autoscaled service tier, shared state and cache pulled out into Redis, the database behind it, and a queue draining into workers for everything off the request path:

![Scalable backend architecture: clients hit a load balancer that spreads requests across an autoscaled pool of stateless service instances; instances read and write a shared Redis (cache and sessions) and the database, and enqueue background jobs onto a message queue drained by a pool of workers that also write to the database](https://mermaid.ink/img/Zmxvd2NoYXJ0IExSCiAgICBDbGllbnRbQ2xpZW50c10gLS0+IExCW0xvYWQgQmFsYW5jZXJdCiAgICBzdWJncmFwaCBUaWVyW1N0YXRlbGVzcyBzZXJ2aWNlIHRpZXIgLSBhdXRvc2NhbGVkXQogICAgICAgIFMxW0luc3RhbmNlXQogICAgICAgIFMyW0luc3RhbmNlXQogICAgICAgIFMzW0luc3RhbmNlXQogICAgZW5kCiAgICBMQiAtLT4gUzEKICAgIExCIC0tPiBTMgogICAgTEIgLS0+IFMzCiAgICBTMSAtLT4gQ2FjaGVbKFJlZGlzOiBjYWNoZSArIHNlc3Npb25zKV0KICAgIFMxIC0tPiBEQlsoRGF0YWJhc2UpXQogICAgUzEgLS0+fGVucXVldWUgam9ic3wgUVtbTWVzc2FnZSBxdWV1ZV1dCiAgICBRIC0tPiBXMVtXb3JrZXJdCiAgICBRIC0tPiBXMltXb3JrZXJdCiAgICBXMSAtLT4gREI=?type=png&bgColor=FFFFFF)

Trace a request: it hits the **load balancer**, which sends it to any healthy **instance** (they are identical, so it does not matter which). That instance reads the user's session and hot data from **Redis**, does the fast part of the work against the **database**, drops anything slow onto the **queue**, and returns. **Workers** pick up the queued jobs and finish the slow work out of band. Under a traffic spike the autoscaler adds instances and the queue soaks up the backlog; when an instance dies the load balancer routes around it and a fresh clone takes its place. Nothing in the request path is a bottleneck that a copy cannot relieve - which is the definition of a scalable backend.

## The order you build it in

1. **One server**, vertically scaled - fine to start.
2. **Make it stateless** (externalize sessions), run **multiple instances behind a load balancer**.
3. **Autoscale** on load so capacity follows demand.
4. **Add a cache** in front of expensive reads.
5. **Move non-critical work to a queue** with worker pools.
6. **Add the resilience patterns** (rate limiting, timeouts, circuit breakers) before you actually need them.
7. **Split services** only when a real boundary forces it, behind an API gateway.

Climb it in order and the backend rarely becomes the wall. What usually does is the tier behind it - the database - which is the next and hardest post in this set.