Design a Rate Limiter | System Design Question

"Design a rate limiter" is one of the most common warm-up system design questions, and it is deceptively deep. It looks like "just count requests," but a good answer walks through picking an algorithm, choosing where the counter lives, and then defending the design against concurrency and a distributed deployment. Let us take it end to end the way you would in a 60-minute interview, using the 4-step framework: scope it, sketch a high-level design, go deep, then wrap up.

A rate limiter controls how much traffic a client can send in a window. If a caller exceeds the limit, the extra requests are rejected (usually with an HTTP 429). Real APIs run this everywhere: Twitter capped tweets at 300 per 3 hours, Google Docs' read API allows 300 per user per 60 seconds, and every large API you have ever hit throttles you somewhere. It protects you from denial-of-service floods, keeps a paid third-party API bill from exploding, and stops one noisy user from overloading shared servers.

Step 1 - Scope the problem

Do not start designing. Rate limiting has half a dozen valid shapes, and the interviewer is holding the constraints. Start asking. Here is roughly how that conversation goes, including the moments where the interviewer does not hand you a clean answer:

You: Are we building a client-side limiter or a server-side one? Interviewer: Server-side, for our API.

You: Do we throttle by IP, by user ID, or something else? Interviewer: Make it flexible. Different rules for different things.

You: What scale are we targeting? Rough peak requests per second? Interviewer: You tell me. Assume we are a large product.

You: Okay, I will assume 1 million daily active users and design for a peak around 10,000 requests per second. I will write that down and we can revisit it. Interviewer: Fine.

You: Is this a single server or a distributed deployment? Interviewer: Distributed. Many app servers.

You: Should the limiter be its own service, or middleware inside the app? Interviewer: Your call. Justify it.

You: And when someone is throttled, do we tell them clearly? Interviewer: Yes. Also, can it double as our fraud-detection and bot-scoring system? You: We can feed its data into that later, but real fraud detection is a separate system with its own models and storage. If I build it into the limiter I bloat the hot path that every request goes through. I will keep the limiter focused on counting and defer fraud scoring.

Notice the three patterns, because they show up in every scoping round:

• A clean answer ("server-side", "distributed") - use it directly.
• A punt ("you tell me the scale") - propose a concrete number, say it out loud, write it down as a stated assumption, and move on. Never freeze because they did not give you a number.
• A curveball where the interviewer bolts on scope (fraud detection) - name its one-line cost and defer it, rather than silently expanding the problem.

If you tend to blank when the problem lands, that recovery move is the whole game - I wrote about it separately in what to do if you go blank.

Requirements

Now write the requirements as tight bullets, split into functional and non-functional (the distinction matters and interviewers watch for it):

Functional

• Accurately limit excessive requests against a configurable rule.
• Rules are flexible: per user, per IP, per API endpoint, each with its own limit and window.
• A throttled client gets a clear signal (HTTP 429) and enough header info to back off.

Non-functional

• Low latency: the limiter sits on every request, so it must add almost no time.
• Low memory: track counters cheaply, not one heavy record per request.
• Distributed: the limit is shared correctly across many app servers.
• Highly available and fault tolerant: if the limiter's datastore hiccups, the whole API must not go down with it.
• Correct under concurrency: no double-counting when requests race.

A quick estimate

At 10,000 requests per second, the limiter does one or two counter operations per request, so call it 10k-20k datastore ops per second. A single Redis node handles well over 100,000 simple ops per second, so throughput is not the worry - latency and correctness are. Memory is tiny: a counter is a key plus a small integer and a TTL. Even with a few million distinct (user, endpoint) keys alive at once, that is on the order of tens to low hundreds of MB. The takeaway to say out loud: this is cheap to store, so the design pressure is all about where the counter lives and how we update it safely.

Step 2 - High-level design

I will build this up piece by piece rather than draw the finished box diagram cold.

Where does the limiter run? Three options. A client-side limiter is a non-starter: clients can be forged or patched, and I do not control third-party callers, so any limit there is advisory at best. A server-side check inside each API handler works but scatters the same logic across every endpoint and every service. The cleanest option is a rate-limiter middleware that sits in front of the API - every request passes through it before reaching business logic, and the app code stays clean. If the system already uses microservices with an API gateway (which typically does auth, SSL termination, and IP allowlisting anyway), the limiter is a natural feature to add there. I will go with the middleware/gateway approach and justify it that way: one place, every request, no duplicated logic.

Where does the counter live? The limiter needs to read-and-update a count on every request. A relational database is out immediately - disk access is far too slow for something on the hot path of every call. That points at an in-memory store. Two candidates: Memcached and Redis. Both are fast, but Redis gives me atomic INCR and EXPIRE, plus sorted sets and Lua scripting, which (as we will see in the deep dive) are exactly the primitives that make concurrent and sliding-window limiting correct. So: Redis, with INCR to bump the counter and EXPIRE to auto-reset it at the end of each window - which also means expired counters clean themselves up, keeping memory low. Redis' own docs describe this INCR/EXPIRE rate-limiting pattern directly.

Where do the rules come from? Limits like "5 login attempts per minute" are configuration, not code. They live in config files on disk, and a background worker periodically loads them into a cache so the middleware can look them up without touching disk on every request. Lyft's open-source ratelimit component models rules exactly this way - a domain with descriptors and a requests_per_unit.

Chaining those decisions together gives the high-level picture:

The flow: request hits the middleware, middleware reads the matching rule from the cache and the counter from Redis, and either forwards the request (and increments the counter) or rejects it with a 429.

Step 3 - Deep dive

The high-level design hides the interesting decision: which counting algorithm? This is where you reason past "just increment a counter," because the naive counter has a real flaw. There are five classic algorithms - a good rundown with production data is in this 2026 engineering reference - and I will walk the ones that matter and land on a choice.

Token bucket

A bucket holds up to N tokens. A refiller adds tokens at a fixed rate up to the cap; overflow is discarded. Each request must take one token to pass; if the bucket is empty, the request is dropped.

It takes two parameters: bucket size and refill rate. Its best property is that it allows short bursts - a client that was quiet can spend a full bucket at once, which matches how real traffic behaves. It is memory efficient (two numbers per key: token count and last-refill timestamp). This is why Stripe and AWS API Gateway both use it. The one downside is that the two parameters can be fiddly to tune.

You do not literally run a background timer adding tokens. You compute tokens lazily on each request from the elapsed time since the last one:

type Bucket = { tokens: number; lastRefill: number };

class TokenBucket {
  private buckets = new Map<string, Bucket>();

  constructor(
    private capacity: number,   // max tokens
    private refillPerSec: number // tokens added per second
  ) {}

  allow(key: string): boolean {
    const now = Date.now();
    const b = this.buckets.get(key) ?? { tokens: this.capacity, lastRefill: now };

    // Add the tokens that accrued since we last looked, capped at capacity.
    const elapsedSec = (now - b.lastRefill) / 1000;
    b.tokens = Math.min(this.capacity, b.tokens + elapsedSec * this.refillPerSec);
    b.lastRefill = now;

    let allowed = false;
    if (b.tokens >= 1) {
      b.tokens -= 1; // spend a token
      allowed = true;
    }

    this.buckets.set(key, b);
    return allowed;
  }
}

In production the Map is Redis and the read-modify-write must be atomic - more on that below.

Leaking bucket

Same bucket idea, but requests are added to a FIFO queue and drained at a fixed outflow rate; if the queue is full, new requests are dropped. It produces a smooth, constant output rate, which is good when a downstream system needs steady load. Shopify uses a leaky bucket for its API. The downside is the mirror image of token bucket's strength: it does not allow bursts, and a queue full of old requests can starve fresh ones.

Fixed window counter

Divide time into fixed windows (say, one minute), keep one counter per window, increment on each request, and reject once the counter passes the limit. Cheap and easy to understand - it is the natural INCR/EXPIRE implementation. But it has a real correctness bug at the window edges:

Because the counter resets hard at the boundary, a client can fire the full quota just before the reset and the full quota just after, pushing double the limit through a single rolling minute.

Sliding window log

Fix the edge problem exactly: store the timestamp of every request in a sorted set (Redis sorted sets are the standard tool - see this rolling rate limiter writeup). On each request, drop timestamps older than the window, add the new one, and accept only if the set size is within the limit. It is perfectly accurate for any rolling window. The cost is memory: you store one entry per request, and even rejected requests can sit in the log, which gets expensive under heavy traffic.

Sliding window counter

The hybrid that most people reach for in production. Keep the fixed-window counters, but smooth the edge by weighting the previous window by how much of it still overlaps the current rolling window:

The estimate is requests in current window + requests in previous window * overlap fraction. It only needs two counters per key, smooths out the edge spikes, and is close enough to exact that Cloudflare, which runs it at massive scale, measured only 0.003% of requests wrongly allowed or limited across 400 million requests. It assumes the previous window's traffic was evenly spread, which is the only place it can be slightly off.

Picking one

Algorithm	Memory	Bursts	Accuracy	Good for
Token bucket	Low (2 values)	Allowed	Good	General API limiting, bursty traffic
Leaking bucket	Low	Smoothed out	Good	Steady downstream load
Fixed window	Very low	Edge spikes	Weak at edges	Simple quotas, rough limits
Sliding window log	High (per request)	Exact	Exact	Strict, low-volume limits
Sliding window counter	Low (2 counters)	Smoothed	Near exact	High-scale general default

My default answer: token bucket, because it is simple, memory efficient, and its burst tolerance matches real traffic - which is exactly why Stripe and AWS use it. If the interviewer stresses accuracy across a rolling window at high volume, I switch to sliding window counter and cite the Cloudflare number. State the choice and the fallback; that shows you understand the trade-off rather than memorizing one algorithm.

Making it correct in a distributed setup

Now the part that separates a shallow answer from a strong one. Two problems appear the moment there is more than one thread or server.

Race condition. The naive check is: read the counter, test if counter + 1 exceeds the limit, then write it back. Between the read and the write, another request can slip in:

Both requests read 3, both write 4, and one request effectively went uncounted. A lock around the whole read-check-write would fix it but would serialize the hot path and kill latency. The right tools are atomic operations: Redis' INCR is atomic on its own, and for the read-modify-write that token bucket and sliding window need, you wrap the logic in a Lua script (Redis runs it atomically server-side) or use sorted sets. That is the real reason Redis beat Memcached back in the design decision.

Synchronization. The web tier is stateless, so a load balancer can send the same client to different app servers over time. If each server keeps its own local counter, none of them sees the true total, and the shared limit is meaningless. You could pin each client to one server with sticky sessions, but that is neither scalable nor fault tolerant. The clean fix is the one already in the design: a centralized Redis that every limiter instance reads and writes, so there is a single source of truth for each counter.

Putting the distributed picture together:

Telling the client it is throttled

When a request is rejected, return HTTP 429 Too Many Requests (MDN reference). Good clients also want to know the limit and when to retry, which is what rate-limit headers are for. The widely used trio is X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Retry-After; there is also an IETF draft standardizing RateLimit headers that is worth a mention. For some workloads (say, an order that got throttled by an overload spike) you can enqueue the rejected request to process later instead of dropping it.

Follow-up questions to expect

• What if Redis goes down? It is a single point of failure on the hot path, so decide the failure mode explicitly. Fail-open (let requests through when the limiter cannot check) keeps the API available at the cost of temporarily unenforced limits - usually the right call, since a limiter outage should not take down the product (that was a stated non-functional requirement). Back it with a Redis replica or cluster, and optionally a small local fallback counter.
• A hot key. If one user, IP, or a single global bucket gets hammered, that one Redis key becomes the bottleneck and every request contends on it. Options: shard the key, or give each app server a local token bucket for a slice of the global limit and reconcile approximately - trading a little accuracy for a lot less contention.
• Does the extra Redis hop hurt latency? One in-memory round trip is sub-millisecond, but you can pipeline the read and write, cache rules locally (already in the design), and co-locate Redis with the app tier. For global traffic, run a limiter near each region.
• How does this behave across data centers? A single central Redis is far from distant users. Run a limiter per region and synchronize with eventual consistency - accept that a client hitting two regions might briefly exceed the global limit. That is the same consistency trade-off that shows up everywhere in distributed systems.
• Hard vs soft limits. Hard means the limit is never exceeded; soft allows a brief overshoot. Token bucket's burst tolerance is effectively a soft limit, which is often what you actually want.
• What layer? Everything above is application-level (HTTP, layer 7). You can also limit lower down - for example dropping traffic by IP with iptables at the network layer - as a coarser first line of defense.

Wrap up

The design in one breath: a rate-limiter middleware (or API gateway feature) on every request, a token bucket algorithm by default (sliding window counter when accuracy matters), counters in a centralized Redis updated atomically with INCR/EXPIRE or a Lua script, rules loaded from config by a background worker, and throttled callers answered with HTTP 429 plus rate-limit headers.

The bottleneck is Redis and any hot key; the main failure mode is a limiter outage, which we handle by failing open so the product stays up; and the next scale curve is going multi-region with eventual consistency. If you can build it up in that order - scope, then a store and an algorithm chosen from real numbers, then defend it against races and distribution - you have given a strong answer to one of the most common system design questions there is.

For another end-to-end walkthrough in this style, see design a URL shortener.