Engineering blog: Prexy reverse proxy

Welcome to our first engineering blog. It may be a bit more technical than you are used to from the other blogs, but we did our best to make it understandable for everyone. In this article we are going to talk about Prexy, a new piece of software within Clonable that is used for applying substitution rules.

Background

As a user of Clonable, you may already be familiar with the substitution rule functionality. You can use these search and replace rules to exchange pieces of text or code with a self-selected variant. For example, you can use this to replace an API key or an analytics ID so you can do different analysis for your original site and your translated clones. Internally, these same rules are also used for replacing your original domain name to your clone's domain name and a number of other things.

Since the very first version of Clonable, this functionality has never been updated, which in itself is not surprising, because it did its job just fine. Still, improvements were possible, both in terms of performance and usability. Once in a while, we Clonable take a piece of our product that hasn't been spent for a long time to start improving it. For example, last year we overhauled all of our data infrastructure to make it faster, more scalable, and more fault-tolerant, and before that we also rebuilt the URL translation functionality from the ground up. This quarter it was the turn of the substitution rules.

Old situation and limitations

Substitution rules are applied in Clonable only at the very last stage, just before sending the response back to the client. In the first version of Clonable, we therefore chose to implement this within the NGINX web server using replace-filter-nginx-module, a module created by OpenResty. The module uses sregex as its streaming regex engine, also created by OpenResty. The streaming aspect is important in this case because we don't want to load each response fully into memory. If we did, a number of large responses could cause NGINX to crash or crash because of insufficient memory availability. A second important feature of sregex, is that it can process multiple lines in parallel. This is because each clone already has some standard rules and, in addition, sometimes some rules added specifically for the clone. If these could not be handled in parallel, we would still need to buffer the entire response, because we would need to be able to match it again after the first rule for the second rule.

Last year, however, we increasingly encountered strange performance problems. These problems were often short-lived, but could increase the response time of a page by seconds in extreme cases. After one such spike, the page often loaded normally again, making the problem difficult to reproduce. To debug this further, we added the Clonable-Timings header to all responses, which allowed us to roughly determine which step of the translation process was taking so much time. This revealed that in almost all cases, the upstream responded quickly, and the translation was also fairly smooth. However, there was a gap between completing the translation and fully completing the request, and this gave us a hint that it might have to do with the substitution rules.

The concurrency model of NGINX

However, this did not yet give us an answer to the question of why these delays occurred only sporadically, and why this happened when the servers were not even loaded to half their capacity. To better understand this phenomenon, we need to look a little deeper into how NGINX handles workloads.

NGINX has a worker architecture with one master process and multiple worker processes. Connections are distributed among the workers to handle multiple requests simultaneously. However, this set-up also has a drawback: when a worker is busy processing a request, the other requests that are also assigned to that same worker have to wait. This can cause one heavy request to cause delays for multiple requests, even if the other workers have nothing to do. This effect can be well seen in the image below. During the creation of this image, a stress test is running across 10 different connections. The image then clearly shows that 3 workers are busy, and a fourth is doing almost nothing.

Time for Prexy

With the bottleneck clearly visible, we created a plan to improve it. This project was named Prexy, an amalgamation of RegEx and Proxy. There were 3 main requirements for the end result:

It should be a drop-in replacement for the current setup. The substitution rules must be applied the same way to avoid breaking things in existing setups.
The substitution rules must be streamed to prevent the solution from using too much memory.
The new solution must be faster than the current solution.

First, we had to look for a powerful multi-threaded runtime that could process requests efficiently. After considering several options, the choice landed on the Tokyo runtime for the Rust programming language. Rust is known for allowing you to write fast programs without the safety risks of other low-level languages such as C++. The Tokyo runtime is a flexible asynchronous runtime made for network applications. One of the most important features for us is the fact that it is work-stealing. This means that, unlike NGINX, a request is not bound to one worker, but a worker who has nothing to do can "steal" work from another worker. This way, the resources of our servers can be better utilized.

First prototype

After choosing the underlying technologies, we decided to set up an initial prototype to estimate roughly how much speed gain this project would give us and whether it was worth it at all. As the first implementation for the regex engine (the part that actually processes and applies the rules), we used the standard regex library (or crate as they are called in Rust terminology). This engine, like sregex, is non-backtracking, which means less risk of a ReDoS issue. In a ReDoS, a regular expression faces such an input that the time it takes to evaluate the input increases exponentially. This can be disastrous for performance, and caused cloudflare to have a major outage. So for us it is important that the engine we use is non-backtracking.

Using this regex engine, we built an initial prototype. This prototype used hard-coded rules and the regex engine buffered the entire response instead of streaming it, but this was enough for an initial performance test. Our test setup consisted of two VMs, with one VM running a setup on which both the old solution and the new solution could run. That way we could easily make comparisons between the two methods. The other server ran NGINX which served as the origin server and served a test file. On the same server also ran the tool wrk, with which we generated the load. This is not entirely optimal, since for pure data you prefer to run the load generator on a separate VM, but this VM had enough resources so that NGINX and wrk did not interfere with each other.

The first test results were obvious: Prexy was about 22 times faster in handling a single request (see the image below). This gave the project the final green light, because with such a large difference there was enough room to absorb some performance degradations that might occur with making the functionality complete.

After initial testing, we implemented a number of obvious optimizations, such as caching compiled regular expressions. We also implemented a more efficient way to reuse connections to the upstream. With this we achieved a speed gain of about 20%. We then also tested both solutions under peak load, sending as many requests to the server as possible. Again, the difference was clearly visible and Prexy achieved a throughput about 25x higher than the old solution.

Streaming regex engine

One of the requirements of this project was that the regex engine used be streaming. The prototype's regex create is not, so additional work was needed in this area. However, the regex crate turned out to be highly optimized, so we decided to take this implementation as the basis for our streaming engine, though. When streaming data through a regex engine, there are a few things important to keep in mind. First, you don't know what data is still to come. Therefore, you need to start working with partial matches: matches that are not yet complete, but have already matched 1 or more characters. When finding a complete match, you then have to check that there are no overlapping partial matches, because these can also end up being matches (and a longer match wins over a shorter match in the case of overlap). So you can only start processing a match if all currently partial matches turn out not to be a complete match.

Further optimizations

As expected, rebuilding the regex engine had a negative impact on Prexy's performance. However, there was still plenty of margin compared to the old solution, and by adding further optimizations, we ended up even outperforming before the regex engine rebuild. One of the optimizations we applied was remembering whether to apply a rule to a particular file. Static assets such as CSS and JS files almost never change and for that it is useless to execute a rule every time when it never matches on that particular file. An alternative to this was to cache the result of the replacements, but the disadvantage of this strategy is that it takes a lot of memory to store all the files in. By remembering which rules to apply, we only need a few bytes per file by storing the information as a bitmap.

In addition, we take full advantage of optimizations within the regex engine by, for example, not keeping capturing groups when they are not used in the replacement. As a result, the regex engine only needs to remember the start and end of the entire match, which saves work. We also disabled named capturing groups, since this was quite complex in the streaming variant and since the old solution didn't support that either, it wasn't necessarily necessary. All these optimizations ultimately provided the following performance:

The end result

After testing Prexy extensively to ensure that its behavior was indeed the same as the old solution, we began rolling it out incrementally. Over a period of about a week, we activated Prexy for each clone. In our monitoring, the timing of activation was often easy to see. A decrease of about 50% in response times was not rare to see. Minor differences of about 10% were also seen with other customers who had very few substitution rules. Our own website became about 30% faster (~50ms).

Across our overall infrastructure, we see the following differences after rolling out Prexy.

33% less RAM consumption
20% less CPU usage
10-50 % faster responses

So all in all, we can say that Prexy was a successful project. By replacing an outdated NGINX module, we can now use newer and more efficient techniques that make a clear measurable difference for all our customers. In the future, we will continue to improve Clonable, both in terms of new features and enhancing existing functionality.

Thanks for reading this engineering blog. Let us know if you like to get more technical insights like this into our product more often. Got excited about this project? Then check out our jobs page :)

Engineering Blog - Prexy: super fast search and replace rules