Happy Sunday y’all,
I got a peach of a concept to close out this week. I hope you’re appropriately excited. Most systems design resources overlook this key idea, but this is the best tech interview prep resource for a reason. We built different. Today, we’ll cover the concept of Rate Limiters. Most websites/APIs that deal with high levels of traffic implement Rate Limiters to ensure that their services run smoothly. This post will introduce you to the idea of Rate Limiters and will tell you about the different kinds. That will give you the ability to be able to implement the right kinds when designing APIs/systems for your interviews or work.
What are Rate Limiters?
A rate limiter is a tool that helps you control the frequency of a program's execution or the amount of traffic that it receives. When websites and applications receive surges in traffic, the demand becomes so great that it can result in delays, outages, and crashes. Rate limiters prevent this by only accepting requests at a rate that the website or program can handle. This helps reduce the risk of overuse and/or misuse by malicious actors.
They have several benefits-
They provide safety against DOS/DDOS attacks by limiting the amount of traffic that the application/website/API needs to deal with. Also protects against malfunction/logical error in scripts.
They can help reduce costs by preventing lots of calls at moments when they might be expensive.
An effective way to segment policies/quotas.
Ensures that critical tasks can be handled quickly.
Kinds of Rate Limiters
There are multiple kinds of Rate Limiters. Different kinds of Rate Limiters become useful in different scenarios.
Request Rate Limiter
The simplest kind of Rate Limiter. In these, we just restrict the number of API/system calls that a user can make per second. This simplicity makes it great as a filter against most overloading calls. Pretty much all platforms include this in their pipeline to protect their service.
Concurrent Rate Limiter
Imagine you had APIs that required a lot of processing/time to run. This is common for many Data Science/AI/Machine Learning APIs that run data analytics on the input. In such cases, instead of restricting the number of calls per second, you might want to want to restrict the number of calls that can run at once.
Many cloud-based GPU services, Data Collection services, and other platforms that use parallelism often use Concurrent Rate Limiters as a way to tier different levels of users.
Fleet Usage Load Shedder
Prevention is better than cure. The worst thing that can happen to your service/API would be your high-priority tasks being lost in a sea of lower priority requests. One easy way to prevent this to have a reserved portion of your computing power dedicated solely to high-priority tasks. This portion only gets triggered to do high-priority tasks, when the normal workers are filled. These reserve workers reject all low-priority tasks.
Worker Utilization Load Shedder
In some very rare cases, all your workers will be backed up. In such cases, you have to start dropping the lowest priority tasks. Once the important tasks are taken care of, we can add the lower important tasks into our queue again.
This process must be done slowly and with a great amount of care. If you drop too many tasks too quickly, your system will think that everything is good and add the tasks back in quickly. Then you’ll have a large backlog again, which will be dropped quickly. This cycle will keep happening and will add a lot of unneeded computational overhead. By adding and removing tasks slowly, we avoid this.
Obviously, most good systems/APIs will use multiple kinds of systems all intertwined together. That will allow you to utilize each kind for their benefit.
In the comments below, share what topic you want to focus on next. I’d be interested in learning and will cover them. To learn more about the newsletter, check our detailed About Page + FAQs
If you liked this post, make sure you fill out this survey. It’s anonymous and will take 2 minutes of your time. It will help me understand you better, allowing for better content.
https://forms.gle/XfTXSjnC8W2wR9qT9
Happy Prep. I’ll see you at your dream job.
Rate Limiters are super cool,
Devansh <3
To make sure you get the most out of System Design Sundays, make sure you’re checking in the rest of the days as well. Leverage all the techniques I have discovered through my successful tutoring to easily succeed in your interviews and save your time and energy by joining the premium subscribers down below. Get a discount (for a whole year) using the button below
Reach out to me on:
Instagram: https://www.instagram.com/iseethings404/
Message me on Twitter: https://twitter.com/Machine01776819
My LinkedIn: https://www.linkedin.com/in/devansh-devansh-516004168/
My content:
Read my articles: https://rb.gy/zn1aiu
My YouTube: https://rb.gy/88iwdd
Get a free stock on Robinhood. No risk to you, so not using the link is losing free money: https://join.robinhood.com/fnud75