How Discord Handles Billions of Messages [System Design Sundays]

Something you should definitely know for designing better systems

Oct 17, 2022

To learn more about the newsletter, check our detailed About Page + FAQs

To help me understand you better, please fill out this anonymous, 2-min survey. If you liked this post, make sure you hit the heart icon in this email.

Recommend this publication to Substack over here

Take the next step by subscribing here

Storage is one of the most important components in Systems Design,

Discord is one of the biggest social network platforms out there. The ability to create groups of people that share common interests (servers) is quite powerful. In my university, Discord is one of the most popular social media apps. People use it for everything, from 1-1 interaction to setting up club meetings. Discord is steadily gathering usage and getting mainstream.

Discord had 140 million monthly active users in 2021, a 40% increase on the previous year.
-Source

So how does Discord store the tons of messages users generate? Let’s get into it.

Private text chat heavy Discord servers send a decent number of messages, easily reaching between 100 thousand to 1 million messages a year.

Taken from Discord’s official blog

To those of that want the full details, you can read Discord’s engineering blog answering the same question (linked above). This post will primarily refer to that and the other post, HOW DISCORD INDEXES BILLIONS OF MESSAGES

Key Points-

The preprocessing- To find their needs and requirements, Discord did a large scale analysis on the kind of data generated. You will not be able to do so in an interview context. However you can implement a technique I call implementing the personas. Here you think of the system you are being asked to design and think of the kinds of users that would use it (and how they would do so). Use that to figure out monetization and start implementing the technical details then. This is good because you demonstrate your ability to think in terms of business cases. Also this will allow you to dictate the battle and really pick the areas you’re mosy comfortable with.
Choosing the right database- When stoing things, the first requirement is always the database used. Discord settled on Cassandra after going over their requirements. Their requirements as follows-
How Cassandra Works- “The best way to describe Cassandra to a newcomer is that it is a KKV store. The two Ks comprise the primary key. The first K is the partition key and is used to determine which node the data lives on and where it is found on disk. The partition contains multiple rows within it and a row within a partition is identified by the second K, which is the clustering key. The clustering key acts as both a primary key within the partition and how the rows are sorted. You can think of a partition as an ordered dictionary”
Indexing the messages- Once you start storing a lot of messages, the search functionality becomes the next important step. Indexing is a great tool to search through large datasets efficiently.
Search Needs-
Mode Chosen- Given all these requirements, the team chose ElasticSearch. IBM has a wonderful video going ElasticSearch and its utility so I’m linking that below. It’s a great overview without getting too much into the implementation (which is what most of you need for understanding when to use it). We’ll do a more detailed post on it soon because of how important it is in modern systems.

If you have the time, I would definitely suggest reading through both writeups, because the authors give a lot of details. They mention some of the challenges they went through, the results of running through different tests, etc. As I’ve stated many times, Engineering Blogs are some of the best resources on the internet to up your game quickly. I used them to skyrocket my Machine Learning skills. That journey is covered below-

Technology Made Simple

How to become very good at Machine Learning[Storytime Saturdays]

To learn more about the newsletter, check our detailed About Page + FAQs To help me understand you better, please fill out this anonymous, 2-min survey. If you liked this post, make sure you hit the heart icon in this email. Recommend this publication to Substack over here…

2 years ago · 4 likes · Devansh

I created Technology Interviews Made Simple using new techniques discovered through tutoring multiple people into top tech firms. The newsletter is designed to help you succeed, saving you from hours wasted on the Leetcode grind. I have a 100% satisfaction policy, so you can try it out at no risk to you. You can read the FAQs and find out more here. Scroll down till the end to get 20% off for up to a whole year.

If you have enjoyed this post so far, please make sure you like it (the little heart button in the email/post). I also have a special request for you.

Special Request

This newsletter has received a lot of love. If you haven’t already, I would really appreciate it if you could take 5 seconds to let Substack know that they should feature this publication on their pages. This will allow more people to see the newsletter.

There is a simple form in Substack that you can fill up for it. Here it is. Thank you.

https://docs.google.com/forms/d/e/1FAIpQLScs-yyToUvWUXIUuIfxz17dmZfzpNp5g7Gw7JUgzbFEhSxsvw/viewform

To get your Substack URL, follow the following steps-

Open - https://substack.com/
If you haven’t already, log in with your email.
In the top right corner, you will see your icon. Click on it. You will see the drop-down. Click on your name/profile. That will show you the link.

Click on the circled place to get your URL

You will be redirected to your URL. Please put that in to the survey. Appreciate your help.

In the comments below, share what topic you want to focus on next. I’d be interested in learning and will cover them. To learn more about the newsletter, check our detailed About Page + FAQs

If you liked this post, make sure you fill out this survey. It’s anonymous and will take 2 minutes of your time. It will help me understand you better, allowing for better content.

https://forms.gle/XfTXSjnC8W2wR9qT9

I’ll see you living the dream.

Go kill all and Stay Woke,

Devansh <3

To make sure you get the most out of System Design Sundays, make sure you’re checking in the rest of the days as well. Leverage all the techniques I have discovered through my successful tutoring to easily succeed in your interviews and save your time and energy by joining the premium subscribers down below. Get a discount (for a whole year) using the button below

Get 20% off for 1 year