How Discord fixed hot partitions by moving trillions of messages off Cassandra

Discord published a detailed account of one of the more painful problems in distributed databases: the "hot partition." It's worth reading closely, because the actual fix wasn't the part everyone talks about.

By early 2022, Discord's message-storage Cassandra cluster had grown from 12 nodes in 2017 to 177 nodes, holding trillions of messages. The system had become a source of constant operational pain. As the engineering team put it, "our on-call team was frequently paged for issues with the database, latency was unpredictable, and we were having to cut down on maintenance operations that became too expensive to run."

The root cause was popular channels overwhelming individual nodes. When a lot of users concurrently read the same channel, the requests all land on the same partition. Cassandra has no built-in way to limit or share that concurrent load, so "latency in the node would increase as the node tried harder and harder to serve traffic and fell further and further behind." Java garbage-collection pauses on top of that meant some nodes needed manual reboots to recover.

The headline decision was migrating from Cassandra to ScyllaDB, a Cassandra-compatible database written in C++ with no garbage collector. That alone is a reasonable story: fewer nodes, more disk per node, lower tail latency. Discord went from 177 Cassandra nodes to 72 ScyllaDB nodes, each with 9 TB of disk versus an average of 4 TB before. p99 latency for fetching historical messages dropped from 40-125ms to a steady 15ms, and message inserts went from 5-70ms p99 down to 5ms.

But the database swap alone would not have solved hot partitions. Swapping the storage engine doesn't change the traffic pattern; a popular channel is still a popular channel. The part that actually addresses the root cause is a Rust-based layer of "data services" that Discord placed between its API monolith and the database. These services do request coalescing: if many users ask for the same row at the same time, only one query actually reaches the database, and every requester gets the same result. They also route consistently by channel ID, so retries and duplicate reads for the same hot partition don't multiply against the database.

This is the generalizable pattern, independent of Cassandra or ScyllaDB specifically. Any system with a "hot key" problem, whether that's a viral post, a trending product page, or a popular chat channel, benefits from a layer that deduplicates concurrent reads for the same identifier before they reach the store of record. It's a much smaller, cheaper change than a database migration, and it addresses the traffic pattern directly instead of just buying more headroom.

Where it doesn't generalize as cleanly: request coalescing helps most for read-heavy hot keys with cacheable-ish semantics (recent messages in a channel). It's less useful if every request needs strongly consistent, per-user personalized results, since there's nothing to coalesce. Discord also had the scale (trillions of rows, a dedicated migration effort that got a 3-month plan down to 9 days using a custom Rust migrator) to justify building this layer in-house. For most teams, the lesson isn't "write your own data services layer," it's "look for where your load balancer or cache is missing the concept of an identical concurrent request," which sometimes is solvable with existing tools (single-flight patterns, request deduplication in a cache layer) well before it justifies custom infrastructure.

The full technical writeup, including the migration mechanics that got trillions of rows moved in 9 days, is on Discord's engineering blog: How Discord Stores Trillions of Messages.