Introducing Origin Shield Concurrency Limits and Request Queueing

We originally designed Origin Shield to reduce the load on user's origin servers by acting as a mid-tier caching system. Today, we're taking it to the next level with the new concurrency limit features.

With a few easy clicks, the Origin Shield now allows you to configure the maximum concurrent requests allowed to reach your origin at the same time. Additional requests after this threshold is reached are then either queued or blocked.

While a simple feature at first glance, this can have a significant impact on performance, stability and throughput, especially when serving dynamic content or CPU intensive requests from the origin.

Reduced Concurrency = Bigger Throughput

Let's face it, multitasking is hard, and that doesn't only stand for humans. CPUs can get notoriously slow the higher we increase concurrency. While computers appear to be doing many things at once, that's not really the case and the CPU is actually splitting many small tasks into little pieces of instructions.

The more things that happen simultaneously, the more switching happens between small parts of instructions the CPU need to do and the data it's working with. On top of that, additional locking and work can be caused in an application itself.

To illustrate this, we looked at a benchmark of a CPU intensive application served through Apache server and you can see that the more concurrency increased, the longer each request took to process.

This means that in a case of a spike of requests, despite having some request waiting, we can achieve a significantly higher throughput by controlling concurrency. In this example, by limiting concurrency to 50, we could reduce the maximum wait time of a single request by roughly 75%.

Preventing Concurrency of Death

"Concurrency of Death" is something that we unfortunately see more often that we would like with overloaded origin servers.  

A server only has a set amount of bandwidth capacity available. Once this is reached, concurrent requests start slowing down as the uplink spreads usage between many different connections.

In a state of a server where it was already unable to serve requests in time, this means further load is added and as more and more requests fly in, more things slow down until eventually nothing is moving at all. Usually, the only real solution to this ends up being to completely restart the server, break existing connections and allow fresh traffic to flow through, but in extreme cases, even that's not enough.

We're excited to say that the new concurrency controls can help to greatly alleviate this problem. While not a perfect solution, by limiting the number of concurrent requests, the Origin Shield can now make sure that requests are flowing through at a controllable rate. Even if some requests might end up stuck in a queue for a bit or eventually rejected, this prevents the origin from being spiraled into an uncontrollable crawl.

Concurrency Metrics

A big part of being able to control concurrency is also understanding what's actually going on. To help with that, we're happy to introduce the new concurrency metrics that visualize the exact maximum of concurrent and queued requests in the past month. By understanding what's going on, we believe this will be greatly helpful to tune settings for each individual configuration and allow you to achieve optimal results.

Helping make the internet hop!

The new Concurrency Limits are now available for all of our users as a preview release completely free of charge and can be configured under the Origin Shield settings with just a few clicks.

We are thrilled to be able to help developers make the internet hop faster and hopefully minimize unpleasant situations from affecting your backend systems with the new and upgraded Origin Shield.

If you enjoy innovation, working on next-generation cloud technologies and would like to drive a mission of helping build a faster internet, check out our careers page! bunny.net started as a small team, but are growing quickly with ambitious goals and exciting products lined up for the future.