Concepts you should know about system designing (part 1)

You cannot follow a "design Twitter" walkthrough if half the words in it are unfamiliar, and you cannot defend a design if you only know the words and not the mechanics behind them. This post explains the high-level design concepts properly - not a glossary, but how each one actually works, what it costs, how it fails, and when you reach for it. Read it once end to end, then keep it as the reference you check while working through problems.

This is Part 1 of a two-part concepts reference in the System Design Guide series. Part 1 is the HLD vocabulary (architecture and scale); Part 2 is low-level design (SOLID, DRY, design patterns). It is long on purpose - depth is the point.

Scaling: up vs out, and why "out" changes everything

There are two ways to handle more load, and the choice reshapes your entire architecture.

Vertical scaling (scale up) means giving one machine more resources: more CPU cores, more RAM, faster disks. Its appeal is that nothing about your code changes - a bigger box runs the same single-process app. Its problems are fatal at scale: there is a hardware ceiling (you cannot buy an infinitely large server), the cost curve is exponential near the top end, and that one machine is a single point of failure - when it dies, everything dies. Vertical scaling is the right first move for a young system, and often for the database tier longer than people admit, because a single big database is dramatically simpler than a distributed one.

Horizontal scaling (scale out) means adding more machines and splitting work across them. This is how large systems grow, and it has effectively no ceiling. But the moment you have more than one machine, you inherit the entire discipline of distributed systems: you must distribute requests (load balancing), you must decide where each piece of data lives (partitioning), and you must reconcile copies that can disagree (replication and consistency). Every other concept in this post exists because of horizontal scaling.

The pivot that makes horizontal scaling practical is statelessness. A stateless service keeps no client-specific data in its own memory between requests - no session stored locally, nothing that ties a user to a particular instance. Because any instance can serve any request, you can add instances, kill instances, and load-balance freely without users noticing. State does not disappear; you push it down into a shared store (a database or a cache like Redis) and out to the client (a token, a cookie). The rule of thumb: keep your application tier stateless so it scales trivially, and concentrate the hard state problems in the data tier where you can reason about them.

Two words you must not blur: latency is how long a single request takes (measured in milliseconds, and you care about the tail - p99, not just the average); throughput is how many requests you complete per second. They are different axes and you optimize them with different tools. Batching improves throughput but usually worsens latency; a cache improves both. When a requirement says "feels instant," that is a latency (p99) requirement. When it says "10 million users," that is a throughput requirement. Pin down which one you are being graded on.

Distributing traffic: load balancers, proxies, gateways

A load balancer sits between clients and your fleet of servers and spreads requests across them. But how it spreads them matters, and interviewers ask:

• Round robin: hand each new request to the next server in rotation. Simple, and fine when all servers and all requests are roughly equal.
• Weighted round robin: give beefier servers a larger share.
• Least connections: send the next request to the server currently handling the fewest. Better when requests vary wildly in duration, because round robin can pile long requests onto one unlucky server.
• IP hash / consistent hashing on a key: always route a given client (or key) to the same server. Useful when a server holds some cache or session for that client.

Load balancers operate at two layers. An L4 (transport) load balancer routes on IP and port without looking at the payload - fast, protocol-agnostic, but dumb about content. An L7 (application) load balancer parses HTTP and can route on path, headers, or cookies (send /api/* to one pool, /images/* to another), do TLS termination, and cache - more powerful, slightly more expensive per request. It also runs health checks: it periodically pings each server and stops routing to any that fail, which is how a dead instance stops receiving traffic within seconds.

A reverse proxy is the general category the load balancer belongs to: a server that faces clients and forwards to backends, often adding TLS termination, compression, and caching along the way. (Contrast a forward proxy, which faces the other way and represents clients to the internet.) An API gateway is a reverse proxy specialized for microservices: it is the single front door that handles routing to the right service, authentication, rate limiting, request validation, and sometimes response aggregation - so that fifty microservices do not each reimplement auth and throttling. The trade-off is that the gateway is now a critical shared component you must make highly available.

Caching: the biggest lever, and the hardest to keep correct

Caching means keeping a copy of data somewhere faster and closer than its source of truth, so most reads never touch the slow, expensive database. It is the single highest-impact tool in system design because real workloads are enormously skewed - a tiny fraction of items get the overwhelming majority of reads (the hot set), and serving those from memory instead of disk can cut read latency from tens of milliseconds to well under one, while removing most of the load from your database.

Caches live at every layer, and a request may pass through several:

• Client / browser cache: the fastest possible - the data never leaves the device.
• CDN: an edge cache near the user for static assets (more below).
• Application cache: an in-memory store like Redis or Memcached, shared by your app servers, holding hot query results, objects, or computed values.
• Database cache: the DB's own buffer pool keeping hot pages in RAM.

The important design decisions are how you write to a cached system and how you evict.

Write strategies (the order of operations, and what breaks):

• Cache-aside (lazy loading): the application is in charge. On a read, it checks the cache; on a miss it reads the database, writes the value into the cache, and returns it. On a write, it updates the database and invalidates (or updates) the cache entry. This is the default pattern. Its weakness is a race: two clients can interleave a read-miss and a write and leave a stale value cached, so you usually pair it with short TTLs.
• Write-through: every write goes to the cache and the database synchronously, together. Reads are always warm and consistent, but writes pay the latency of both stores, and you cache data that may never be read.
• Write-back (write-behind): writes go to the cache immediately and are flushed to the database asynchronously later. Writes are very fast and you can batch them, but a cache failure before the flush loses data - only acceptable when some loss is tolerable.
• Write-around: writes go straight to the database, bypassing the cache; the cache is populated only on read misses. Good for write-heavy data that is rarely read back soon.

Eviction decides what to drop when the cache is full: LRU (evict the least recently used - the common default, good for temporal locality), LFU (evict the least frequently used - better when popularity is stable), or a plain TTL (every entry expires after a fixed time, which also bounds staleness).

The genuinely hard problem is invalidation - ensuring the cache does not keep serving a value after the underlying data has changed. Get it wrong in one direction and users see stale data; get it wrong in the other and you invalidate too aggressively and lose the cache's benefit. There is a reason the industry joke is that the two hardest things in computer science are cache invalidation and naming things. Related failure modes worth naming: a cache stampede (a hot key expires and thousands of requests hammer the database simultaneously to recompute it - mitigated with locks or staggered TTLs) and the difference between a cache hit and a cold cache after a restart (which can briefly crush the database until it warms).

A CDN (Content Delivery Network) is a globally distributed cache for static content - images, video, CSS, JS - stored at edge locations physically near users. When a user in Mumbai requests an image, they get it from an edge node in or near Mumbai instead of from an origin server in Virginia, cutting hundreds of milliseconds of round-trip time. The CDN fetches from your origin on the first miss and serves the cached copy for everyone after. Any design that serves media at scale puts a CDN in front of object storage.

Databases: choosing the store and making it scale

SQL (relational) databases store data in tables with a fixed schema and relationships, speak SQL, support joins, and offer ACID transactions - Atomicity (all-or-nothing), Consistency (constraints always hold), Isolation (concurrent transactions do not corrupt each other), and Durability (committed data survives crashes). You reach for SQL whenever correctness and relationships dominate: payments, orders, inventory, anything where a half-applied update is unacceptable. Postgres and MySQL are the workhorses.

NoSQL is an umbrella for stores that relax parts of the relational model to gain scale, flexibility, or a specific access pattern. The families:

• Key-value (Redis, DynamoDB): a giant distributed hash map. O(1) lookups by key, no joins. Perfect for sessions, caches, and simple lookups.
• Document (MongoDB): stores JSON-like documents; flexible schema, good when each record is a self-contained object.
• Wide-column (Cassandra, Bigtable): rows with dynamic columns, optimized for enormous write volume and queries along a known partition key.
• Graph (Neo4j): nodes and edges, for data whose value is in the relationships (social graphs, recommendations).

The honest interview answer is rarely "SQL vs NoSQL" as a religious choice; it is "this access pattern, at this scale, with these consistency needs, points to this store." Say why.

Indexing is how a database avoids scanning an entire table to answer a query. An index is a separate, sorted data structure - almost always a B-tree (or B+ tree) - that maps a column's values to the rows containing them, turning an O(n) scan into an O(log n) lookup. The cost is real: every index must be updated on every insert, update, and delete (write amplification), and indexes consume space. A composite index on (last_name, first_name) accelerates queries that filter by last_name or by both, but not by first_name alone (the leftmost-prefix rule). A covering index includes every column a query needs, so the database answers it from the index without touching the table at all. Rule of thumb: index the columns you filter and sort on, and no more.

Normalization vs denormalization is the write-simplicity vs read-speed trade-off. Normalized schemas store each fact once and reference it (no duplication, clean and safe writes, but reads need joins). Denormalized schemas duplicate data so a read hits one place (fast reads, but every copy must be updated on write, risking inconsistency). Read-heavy systems denormalize deliberately - a precomputed timeline is denormalization taken to its conclusion.

Replication keeps copies of your data on multiple nodes for two reasons: to scale reads and to survive a node dying. The dominant model is leader-follower (also called primary-replica): all writes go to one leader, which streams its changes to one or more followers that serve reads. This scales reads beautifully, but introduces replication lag - a follower is briefly behind the leader. That lag causes a concrete bug: a user writes data, is then served by a lagging follower, and does not see their own change (the "read-your-writes" problem), fixed by routing a user's reads to the leader for a short window after they write. Replication can be synchronous (the leader waits for followers to confirm before acknowledging the write - safe, but slow, and a stalled follower stalls writes) or asynchronous (the leader acknowledges immediately and replicates in the background - fast, but a leader crash can lose the not-yet-replicated writes). On leader failure, a failover promotes a follower to leader; done carelessly this causes split-brain, where two nodes both think they are leader and accept conflicting writes.

Sharding (horizontal partitioning) is what you do when the dataset itself is too large or too write-heavy for one machine, even with replication: you split the rows across many machines, each holding a subset. The partition key decides which shard a row lives on, and choosing it well is the entire challenge:

• Hash sharding: shard = hash(key) % N. Distributes evenly, but ranges are scattered and changing N reshuffles everything (see consistent hashing next).
• Range sharding: shard by key ranges (A-M on shard 1, N-Z on shard 2). Range queries are efficient, but you risk hot spots if one range is far more active.
• Directory / lookup sharding: a lookup service maps keys to shards, giving flexibility at the cost of an extra hop and a component to keep available.

Sharding's ongoing pain is what it takes away: cross-shard joins and transactions become expensive or impossible, and a hot key (a celebrity's data) can overwhelm its shard no matter how even the overall distribution is. Resharding a live system is one of the genuinely hard operational tasks in the field.

Consistent hashing: making the shard count changeable

Plain hash sharding has a crippling flaw. If you place keys with hash(key) % N across N servers and then add one server (N becomes N+1), the modulus changes for almost every key, so nearly all your data has to move at once - catastrophic for a live system. Consistent hashing fixes this.

Picture the hash space as a ring from 0 to 2^32 - 1. You hash each server to a point on the ring, and you hash each key to a point on the ring; a key belongs to the first server found moving clockwise from the key's position. Now when you add a server, only the keys that fall between the new server and its predecessor move - roughly 1/N of the keys - and everything else stays put. Removing a server likewise only reassigns that server's keys to the next one clockwise. To avoid uneven distribution (a server owning a huge arc by bad luck) each physical server is placed at many points via virtual nodes, which smooths the load and makes removal spread the orphaned keys across many servers instead of dumping them all on one neighbor. This is precisely how Cassandra and DynamoDB decide where data lives, and it is the reason they can grow and shrink the cluster without a full reshuffle.

The CAP theorem and consistency, explained with the actual scenario

The CAP theorem states that a distributed data store can provide at most two of three guarantees: Consistency (every read sees the most recent write, as if there were a single copy), Availability (every request gets a non-error response), and Partition tolerance (the system keeps working even when the network between nodes drops messages). The catch that people miss: in any real distributed system, network partitions will happen, so P is not optional - which means the real choice, at the moment a partition occurs, is between C and A.

Make it concrete. You have three replicas - n1, n2, n3 - holding a value, and the network partitions so n3 cannot talk to n1 and n2. A write arrives for n1:

• If you insist on consistency (a CP system), you cannot let n1 accept the write, because it cannot propagate to n3, and a later read from n3 would return stale data. So you reject or block the write until the partition heals. You sacrificed availability. A bank balance works this way - returning an error is far better than showing two different balances.
• If you insist on availability (an AP system), you let n1 accept the write and serve reads from every node, accepting that n3 is temporarily stale, and you reconcile once the network recovers. You sacrificed consistency (temporarily). A social feed or a "likes" counter works this way - staying up matters more than everyone seeing the exact same number this instant.

PACELC extends this with the honest observation that the trade-off exists even without partitions: if Partition then Availability-or-Consistency, Else (normal operation) Latency-or-Consistency. Even on a healthy network, guaranteeing that every replica agrees before acknowledging a read costs latency. So the deeper truth is that consistency is something you constantly trade against availability and latency, not a one-time switch.

That is why consistency models are a spectrum, not a binary:

• Strong consistency: every read returns the latest write. Simple to reason about, expensive to provide (needs coordination like a quorum or a single leader on the read path).
• Eventual consistency: if writes stop, all replicas converge to the same value eventually. Reads may be stale in the meantime. Cheap and highly available; correct for feeds, DNS, view counts.
• Causal consistency: a middle ground guaranteeing that causally related operations are seen in order (you always see a reply after the message it replies to), while unrelated operations may be seen in any order.

The same trade-off, dressed as database philosophy, is ACID vs BASE. ACID (relational) chooses correctness and strong guarantees. BASE - Basically Available, Soft state, Eventual consistency - chooses availability and scale, and is the mindset behind most AP NoSQL stores. The read/write-splitting pattern CQRS lives on the BASE side: the read model is a denormalized, eventually-consistent projection of the writes.

Finally, quorum is the dial that tunes where on the C-A spectrum a replicated store sits. With N replicas, require W nodes to acknowledge a write and R nodes to agree on a read. If W + R > N, any read is guaranteed to overlap with the latest write on at least one node, giving strong consistency. Example: with N = 3, choosing W = 2 and R = 2 gives 4 > 3 - strongly consistent while tolerating one node down. Choosing W = 1, R = 1 is fast and highly available but only eventually consistent. You are literally turning a knob between the two.

Asynchronous processing: queues, pub-sub, idempotency

Not everything should happen inside the request. When a user posts, the response should return the instant the post is saved; fanning it out to a million followers, sending push notifications, and transcoding the attached video should happen after, off the request path. The tool for that is a message queue.

A queue is a durable buffer between producers (who enqueue work) and consumers (who process it). It decouples the two - producers do not wait for consumers, and either side can scale or fail independently - and it absorbs traffic spikes by letting the queue grow and draining it at a sustainable rate (smoothing). Kafka, RabbitMQ, and SQS are the common choices, and they differ in important ways. Kafka is a distributed, partitioned, append-only log: messages are ordered within a partition, retained for a configured time (so consumers can replay), and consumers track their position with an offset; it excels at high-throughput event streaming. RabbitMQ is a traditional message broker with flexible routing that typically deletes a message once acknowledged; it excels at task queues and complex routing.

The delivery guarantees matter and interviewers probe them:

• At-most-once: every message is delivered zero or one times - never duplicated, but can be lost.
• At-least-once: every message is delivered one or more times - never lost, but can be duplicated (the common, practical default).
• Exactly-once: delivered precisely once - the strongest and the hardest, usually approximated with at-least-once delivery plus idempotent consumers.

Because at-least-once means the same message can arrive twice (a consumer processes a message, crashes before acknowledging, and the queue redelivers it), consumers must be idempotent: processing the same message twice must have the same effect as processing it once. You achieve this with an idempotency key - tag each operation with a unique id, record which ids you have already applied, and skip duplicates. "Charge this card" must never double-charge on a retry; an idempotency key is what prevents it.

Publish/subscribe (pub-sub) generalizes the queue: producers publish events to a topic, and every subscriber gets its own copy and reacts independently. Where a task queue hands each message to exactly one worker, pub-sub broadcasts each event to many interested services - the backbone of event-driven architectures, where a single "order placed" event fans out to inventory, billing, and email services that never call each other directly. A related safety concept is back-pressure: when consumers cannot keep up, a healthy system signals producers to slow down (or sheds load deliberately) rather than letting queues grow without bound until it falls over.

How services communicate

The protocol and API style shape latency, coupling, and how clients fetch data.

• REST over HTTP is the simple, universal, cacheable default: resources at URLs, standard verbs, stateless. Its weakness is over-fetching and under-fetching (a fixed endpoint returns too much or too little, forcing extra round trips).
• gRPC is binary RPC over HTTP/2 using Protocol Buffers: compact, fast, strongly typed, with streaming built in. It shines for internal service-to-service calls where you control both ends; it is less browser-friendly.
• GraphQL lets the client specify exactly the fields it wants in one query, eliminating over/under-fetching for rich, client-driven UIs, at the cost of server complexity and harder caching.

For real-time delivery, plain HTTP request/response cannot have the server push to the client, so you choose among: WebSockets (a single long-lived, full-duplex TCP connection - the right tool for chat, presence, live collaboration); Server-Sent Events (SSE) (a one-way server-to-client stream over HTTP - simpler, good for live feeds and notifications); and long polling (the client makes a request the server holds open until it has data - a fallback that works everywhere but is heavier). Beneath all of these is the network stack itself, where addressing and routing live - see IPv6 vs IPv4.

The push vs pull decision recurs everywhere and is worth internalizing: does the server push new data to clients as it arrives (low latency, but wasted work if the client is not looking, and hard at massive fan-out), or do clients pull by polling (simple and robust, but either stale or wasteful)? This single axis is exactly what separates fanout-on-write (push the post into every follower's precomputed feed) from fanout-on-read (build the feed by pulling from followees at read time) in a news-feed design.

Reliability and availability

Availability is the fraction of time the system is up, quoted in "nines," and the numbers are worth memorizing because they translate to a downtime budget:

99%      (two nines)   -> ~3.65 days/year   of downtime
99.9%    (three nines) -> ~8.76 hours/year
99.99%   (four nines)  -> ~52.6 minutes/year
99.999%  (five nines)  -> ~5.26 minutes/year

Each additional nine is roughly ten times harder and more expensive, because it forces out more and more manual recovery and demands redundancy at every layer. So when a requirement says "highly available," pin the actual target - four nines and five nines are very different systems.

You buy availability with redundancy (run more instances than you strictly need so a failure has spare capacity to absorb it) and failover (automatically shift work from a failed component to a healthy one). The design goal is the elimination of any single point of failure - any one component whose death takes the whole system down. If your load balancer, gateway, or database leader is a lone instance, that is your SPOF, and the fix is to make it redundant (multiple load balancers behind DNS or a floating IP, a standby leader ready to be promoted).

Health checks and heartbeats are the plumbing that makes failover possible: components periodically emit an "I'm alive" signal (or answer a probe), and when the signal stops, the system routes around the dead node and triggers recovery. Rate limiting protects availability from the demand side by capping how many requests a client may make in a window - implemented with algorithms like the token bucket (tokens refill at a fixed rate; each request spends one; empty bucket means throttled - allows bursts) or the sliding window (count requests in the trailing time window) - which shields you from both abuse and accidental thundering herds.

A few more building blocks worth being able to name and explain:

• Bloom filter: a compact, probabilistic set. It is a bit array plus k hash functions; to add an item you set the k bits it hashes to, and to test membership you check those bits. It can answer "definitely not in the set" with certainty and "possibly in the set" with a tunable false-positive rate, using a tiny fraction of the memory a real set would need, and it never gives a false negative. Web crawlers use one to cheaply skip URLs they have already seen; databases use them to avoid disk lookups for keys that are absent. See Bloom filter.
• Consensus: the problem of getting distributed nodes to agree on a single value (or on who the leader is) despite failures. You will not implement Raft or Paxos in an interview, but knowing that leader election and replicated logs are solved problems with names lets you wave at them credibly ("a consensus protocol like Raft elects the new leader").
• Write-ahead log (WAL): before applying a change, append it to a durable log first; on crash, replay the log to recover. It is the mechanism behind database durability and much of replication.

Storage tiers and delivery

Not all storage is a database. Object (blob) storage - S3 and its equivalents - stores large, unstructured files (images, video, backups, logs) as objects addressed by a key, cheaply and with effectively unlimited scale and high durability. The critical pattern for it is to keep the bytes off your application servers: instead of a client uploading a video through your API (which would saturate your servers' bandwidth and memory), your API hands the client a short-lived, signed URL and the client uploads directly to object storage - which is exactly what presigned URLs are for, and how any media-heavy design avoids melting its app tier. Reads come back out through a CDN.

For completeness, the three storage shapes and when each fits: block storage (raw disk volumes attached to one machine - what databases run on, lowest-level and fastest), file storage (a shared hierarchical filesystem multiple machines can mount - shared documents, legacy apps), and object storage (flat key-to-blob, web-scale, the default for anything user-facing and large).

Putting it together

No real design uses one of these in isolation; a system is a deliberate stack of them chosen to satisfy specific requirements. A read-heavy social application, for instance, might terminate TLS at an L7 load balancer, spread requests across a stateless application tier, serve hot reads from a Redis cache and static media from a CDN over object storage, keep the source of truth in a sharded SQL database with leader-follower replication (reads from followers, writes to the leader), push non-critical work (fanout, notifications, transcoding) onto a Kafka queue consumed by idempotent workers, and accept eventual consistency on the feed while demanding strong consistency on anything involving money. Every problem you will practice is a different combination of these same blocks - and knowing the mechanics, not just the names, is what lets you justify each choice when the interviewer pushes.

Part 2 does the same depth for low-level design - the object-oriented principles (SOLID, DRY) and the design patterns that decide whether your machine-coding solution is clean or a mess. After that, back-of-the-envelope estimation turns a problem's scale requirements into the concrete numbers that tell you which of these blocks you actually need.