Benchmarking Blackwall Against Anubis

This article is under construction.

Table of contents

In Keeping Scrapers Out I talked about Anubis and Blackwall. Anubis is a reverse proxy that protects websites from AI scrapers with a proof-of-work challenge, while Blackwall is my re-implementation of it in Rust. Go is an excellent choice for a reverse proxy, so just writing it in Rust may not have provided noticeable performance improvements in and of itself. Before we start, I want to remark that Anubis is a battle-tested, consciously developed project that I am a fan of. It does its job, it does it well, and it doesn’t need to be faster. Still, it is always a good exercise to try and make a more performant and efficient version, an exercise from which we all benefit.

This article walks through the benchmarks I’ve run comparing Blackwall to Anubis and the results. I’ve done my best to categorize the differences into four kinds: language, tooling, design, and configuration. Knowing what is which is necessary to measure things properly and draw meaningful conclusions.

  • Language: pure Rust vs Go. The intrinsic characteristics of each language.
  • Tooling: the libraries each implementation uses. Tokio falls a bit in between language and tooling, so not every categorization is a clear-cut.
  • Design: intrinsic design choices made by either Anubis developers or me. E.g. internal data structures.
  • Configuration: setting differences. I’ve done my best to keep this to a minimum to keep the results meaningful. The number of connections each process can keep open, active policies…

TLDR

  • Blackwall is faster. Most performance benchmarks I ran favored Blackwall over Anubis by a margin. Memory management gets more interesting though: while Blackwall uses less in most scenarios, in specific scenarios with big payloads Blackwall can hoard memory that Anubis releases. The memory benchmark section goes into more detail.
  • Increasing the number of vCPUs narrows the gap. After running tests with 1, 2, and 4 vCPUs, the gap between Blackwall and Anubis slowly narrowed as the core count increased. I suspect the gap will stabilize as the core count increases, based on the performance characterization of the Go runtime I did in my MSc thesis. The number of vCPUs in my testing rig is limited, so I have yet to confirm this suspicion.
  • The language is not everything. Tooling and design do a lot. For example, the regular expression matching libraries each project uses have made a big difference. So has having buffer pools for the reverse proxy. The biggest takeaway from this is that Anubis has the potential to become faster and more efficient.
  • Anubis has some wins too. For example, it frees memory more quickly than Blackwall in specific scenarios, like when receiving large payloads. This is most likely caused by glibc’s default allocator, which Blackwall uses by default, and can be mitigated by either using a different one as these benchmarks show.

Methodology

Most benchmarks follow a similar formula: a load generator sends a certain number of requests per second to the filter (either blackwall or anubis), which evaluates it based on policy rules and responds accordingly. Then the load generator, filter, and website perform the respective policy evaluation outcome, and the results are recorded. Results can include latency, CPU or memory usage, some profiling data… The load generator can “crank up the heat” gradually, like increasing the number of requests per second every few seconds, to find the breaking point of the filter. You can find the benchmark code and results in the repository.

The load generator cranks up the heat until the filter capacity is reached, which I’ve arbitrarily defined as the point where more than 1% of requests take over 10 ms: the (in)famous P99 is 10 ms. Not all tests follow this, because it wouldn’t make sense for them to, but it is the default on many of them.

Load generatorBlackwall or Anubis Website Results Challenge Policy rules Reverse proxy CHALLENGEALLOWN req/s page,solutionallowedCPU time and memory(every run), CPUprofile (some runs)latencyp99 ≤ 10 ms:raise N

Hardware

My dear old Kelpie: A 10-year-old PC with an Intel Core i7-6700K CPU, 32GB of DDR4 RAM, and as many SSDs and HDDs as I could fit into it. It’s been running the same Linux (Arch btw) for many years now, it’s seen the world change and changed along with it. Now that it runs Wayland instead of X11, it will continue to do so but with fractional scaling which I really appreciate. Back to our topic, the CPU has 4 physical cores and 8 logical cores, which is very relevant to some of our benchmarks. The clock speed was fixed for the duration of the benchmarks: performance mode, no turbo boost. Each process ran pinned to its CPU core to keep them from interfering with each other. This means: the load generator had its own core(s), the website had its own core, and the filters (again, Anubis and Blackwall) had their own core(s).

Configuration

Blackwall has very limited configuration compared to Anubis, but I’ve done my best to keep them as similar as possible. Logging is disabled in both, they use the same signing key, policies, retention settings… And both have challenge difficulty set to 1. This is important because the average time to solve a challenge of the same difficulty is not equal between the two, so setting it to one makes it negligible (around 1 µs).

How to interpret the results

Many of the benchmarks share a few common factors:

Rules

Rules tell the filter what to do with incoming requests. They are defined in policies, and each benchmark uses the same for Anubis and Blackwall. Stock rules are the ones they use by default (again, same for both), and minimal rules are very simple rules used to keep evaluation as simple and lightweight as possible.

Scenarios

Scenarios define what will happen with the requests sent by the load generator and received by the filter.

  • Allowed lets a GET request through directly to the website, which responds with a payload of an arbitrary size.
  • Denied rejects the request directly.
  • Challenge has the filter challenge the request, which forks into the following branches:
    • Solved: the requester solves the challenge and is allowed through.
    • Flood: the requester ignores the challenge and doesn’t reply. Standard DoS scenario.
  • Returning visitor: the requester would have been challenged, but it has receipts (a cookie showing they already solved one) and is allowed through.
  • Upload: similar to allowed, but it is a POST request that carries the payload. Allowed sends a payload from the website to the client, while Upload sends a payload from the client to the website.

Measurements

There are two main measurements to keep in mind when interpreting the results:

Capacity

The rate at which the server can handle requests, for example, 1000 requests per second [req/s].

CPU per request

The time the server spends handling each request. Evidently correlated to capacity. The CPU per request changes significantly depending on request type, so know which scenario we’re running is important. An allow is faster than a challenge because it doesn’t need to check cookies or create challenges. A deny is faster than an allow because it doesn’t need to forward the traffic to the website. As a result, scenarios have a big impact on both capacity and CPU per request.

Fair warning

Most of the benchmarks were run assigning one vCPU to the filter. This hurts Anubis somewhat more than Blackwall, because while both multithreaded schedulers are affected (Go’s runtime and Tokio in our case), Go has garbage collection on top of that. The schedulers, queuing and the garbage collector are, in my opinion, the main cause of the gap between Anubis and Blackwall narrowing in truly multithreaded scenarios.

The other thing to keep in mind is that Anubis has more features than Blackwall, such as proper logging or metrics. I’ve done my best to disable these Anubis features to make the comparison fair, but they will inevitably add significant overhead. I’ve done some measuring and based on the benchmark results , I do believe there is potential to implement those same features in Blackwall while maintaining the performance advantage. It wouldn’t be free though: as some of the benchmarks below show, Anubis gets really close to Blackwall in some scenarios, even with the extra baggage of additional features.

Benchmarks

Onto the real stuff! Let’s start with the two measurements mentioned above.

Capacity

To measure capacity I ran the benchmark thrice and picked the median run, a standard practice. Most of the results were consistent across runs, but some had some deviation. When reading the result graph, a “floating” line shows the value of a run that wasn’t the median. For example in the Blackwall - Allowed scenario, the median is 14,000 rps but the best run had 16,000 rps.

Maximum request rate per coreHighest request rate each server sustains on one core with p99 latency at or under 10 ms, per scenario: the median of three runs, with a tick at each run. Bodies are 1 KiB unless marked. Blackwall's denied bar with minimal rules was measured again with a leaner load generator, because the first one was the limit there.BlackwallAnubisEach run020k40k60k80kMaximum rate at p99 ≤ 10 ms (req/s)Blackwall ÷ AnubisStock rulesAllowed7.0×Blackwall — Stock rules, allowed — 14,000 req/s, the median run — Passed 14,000 req/s in all 3 runs, 16,000 in 1 of 314kAnubis — Stock rules, allowed — 2,000 req/s, the median run — Passed 2,000 req/s in all 3 runs2kDenied8.0×Blackwall — Stock rules, denied — 28,000 req/s, the median run — Passed 28,000 req/s in all 3 runs28kAnubis — Stock rules, denied — 3,500 req/s, the median run — Passed 3,500 req/s in all 3 runs3.5kBot flood3.0×Blackwall — Stock rules, bot flood — 6,000 req/s, the median run — Passed 6,000 req/s in all 3 runs6kAnubis — Stock rules, bot flood — 2,000 req/s, the median run — Passed 2,000 req/s in all 3 runs2kChallenge solved4.0×Blackwall — Stock rules, challenge solved — 4,000 req/s, the median run — Passed 4,000 req/s in all 3 runs4kAnubis — Stock rules, challenge solved — 1,000 req/s, the median run — Passed 1,000 req/s in all 3 runs1kReturning visitor5.3×Blackwall — Stock rules, returning visitor — 8,000 req/s, the median run — Passed 8,000 req/s in all 3 runs8kAnubis — Stock rules, returning visitor — 1,500 req/s, the median run — Passed 1,500 req/s in all 3 runs1.5kMinimal rulesAllowed4.8×Blackwall — Minimal rules, allowed — 24,000 req/s, the median run — Passed 24,000 req/s in all 3 runs, 28,000 in 1 of 324kAnubis — Minimal rules, allowed — 5,000 req/s, the median run — Passed 5,000 req/s in all 3 runs5kDenied5.0×Blackwall — Minimal rules, denied — 80,000 req/s, the median run — Passed 80,000 req/s in all 3 runs — Measured again with a leaner load generator80kAnubis — Minimal rules, denied — 16,000 req/s, the median run — Passed 16,000 req/s in all 3 runs16kBot flood1.4×Blackwall — Minimal rules, bot flood — 7,000 req/s, the median run — Passed 7,000 req/s in all 3 runs7kAnubis — Minimal rules, bot flood — 5,000 req/s, the median run — Passed 5,000 req/s in all 3 runs5kChallenge solved1.6×Blackwall — Minimal rules, challenge solved — 4,000 req/s, the median run — Passed 4,000 req/s in all 3 runs4kAnubis — Minimal rules, challenge solved — 2,500 req/s, the median run — Passed 2,500 req/s in all 3 runs2.5kReturning visitor3.0×Blackwall — Minimal rules, returning visitor — 12,000 req/s, the median run — Passed 12,000 req/s in all 3 runs12kAnubis — Minimal rules, returning visitor — 4,000 req/s, the median run — Passed 4,000 req/s in all 3 runs4kLarge bodies, minimal rulesAllowed, 1 MiB1.3×Blackwall — Large bodies, minimal rules, allowed, 1 mib — 2,000 req/s, the median run — Passed 2,000 req/s in all 3 runs2kAnubis — Large bodies, minimal rules, allowed, 1 mib — 1,600 req/s, the median run — Passed 1,600 req/s in all 3 runs1.6kUpload, 1 MiB1.3×Blackwall — Large bodies, minimal rules, upload, 1 mib — 2,000 req/s, the median run — Passed 2,000 req/s in all 3 runs2kAnubis — Large bodies, minimal rules, upload, 1 mib — 1,600 req/s, the median run — Passed 1,600 req/s in all 3 runs1.6kMaximum request rate per coreHighest request rate each server sustains on one core with p99 latency at or under 10 ms, per scenario: the median of three runs, with a tick at each run. Bodies are 1 KiB unless marked. Blackwall's denied bar with minimal rules was measured again with a leaner load generator, because the first one was the limit there.BlackwallAnubisEach run020k40k60k80kMaximum rate at p99 ≤ 10 ms (req/s)Blackwall ÷ AnubisStock rulesAllowed7.0×Blackwall — Stock rules, allowed — 14,000 req/s, the median run — Passed 14,000 req/s in all 3 runs, 16,000 in 1 of 314kAnubis — Stock rules, allowed — 2,000 req/s, the median run — Passed 2,000 req/s in all 3 runs2kDenied8.0×Blackwall — Stock rules, denied — 28,000 req/s, the median run — Passed 28,000 req/s in all 3 runs28kAnubis — Stock rules, denied — 3,500 req/s, the median run — Passed 3,500 req/s in all 3 runs3.5kBot flood3.0×Blackwall — Stock rules, bot flood — 6,000 req/s, the median run — Passed 6,000 req/s in all 3 runs6kAnubis — Stock rules, bot flood — 2,000 req/s, the median run — Passed 2,000 req/s in all 3 runs2kChallenge solved4.0×Blackwall — Stock rules, challenge solved — 4,000 req/s, the median run — Passed 4,000 req/s in all 3 runs4kAnubis — Stock rules, challenge solved — 1,000 req/s, the median run — Passed 1,000 req/s in all 3 runs1kReturning visitor5.3×Blackwall — Stock rules, returning visitor — 8,000 req/s, the median run — Passed 8,000 req/s in all 3 runs8kAnubis — Stock rules, returning visitor — 1,500 req/s, the median run — Passed 1,500 req/s in all 3 runs1.5kMinimal rulesAllowed4.8×Blackwall — Minimal rules, allowed — 24,000 req/s, the median run — Passed 24,000 req/s in all 3 runs, 28,000 in 1 of 324kAnubis — Minimal rules, allowed — 5,000 req/s, the median run — Passed 5,000 req/s in all 3 runs5kDenied5.0×Blackwall — Minimal rules, denied — 80,000 req/s, the median run — Passed 80,000 req/s in all 3 runs — Measured again with a leaner load generator80kAnubis — Minimal rules, denied — 16,000 req/s, the median run — Passed 16,000 req/s in all 3 runs16kBot flood1.4×Blackwall — Minimal rules, bot flood — 7,000 req/s, the median run — Passed 7,000 req/s in all 3 runs7kAnubis — Minimal rules, bot flood — 5,000 req/s, the median run — Passed 5,000 req/s in all 3 runs5kChallenge solved1.6×Blackwall — Minimal rules, challenge solved — 4,000 req/s, the median run — Passed 4,000 req/s in all 3 runs4kAnubis — Minimal rules, challenge solved — 2,500 req/s, the median run — Passed 2,500 req/s in all 3 runs2.5kReturning visitor3.0×Blackwall — Minimal rules, returning visitor — 12,000 req/s, the median run — Passed 12,000 req/s in all 3 runs12kAnubis — Minimal rules, returning visitor — 4,000 req/s, the median run — Passed 4,000 req/s in all 3 runs4kLarge bodies, minimal rulesAllowed, 1 MiB1.3×Blackwall — Large bodies, minimal rules, allowed, 1 mib — 2,000 req/s, the median run — Passed 2,000 req/s in all 3 runs2kAnubis — Large bodies, minimal rules, allowed, 1 mib — 1,600 req/s, the median run — Passed 1,600 req/s in all 3 runs1.6kUpload, 1 MiB1.3×Blackwall — Large bodies, minimal rules, upload, 1 mib — 2,000 req/s, the median run — Passed 2,000 req/s in all 3 runs2kAnubis — Large bodies, minimal rules, upload, 1 mib — 1,600 req/s, the median run — Passed 1,600 req/s in all 3 runs1.6k

The highest rate each server handled on one core while keeping its p99 under 10 ms. Each bar is the middle of three runs.

The results here are interesting. The Blackwall/Anubis capacity ratio, shown to the right of the bars, changes significantly across scenarios. The best for Blackwall are, in general terms, any of the “stock rules” scenarios and the scenarios that don’t involve issuing new challenges. Because the capacity gap for a scenario widens with stock rules as compared to minimal rules, we can infer that rule evaluation is faster in Blackwall. Scenarios that involve issuing new challenges, on the other hand, are much closer in the “minimal rules” scenarios; which means challenge issuance performance is very close in both Blackwall and Anubis.

In a simplified view, a challenge scenario involves: receiving a request, evaluating it against the rules, creating a new challenge, storing it in memory, returning it to the client, receiving a solution, validating it, and issuing a cookie. The last three steps are skipped in bot flood. Because rule evaluation is mostly trivialized in this scenario, the performance gap of the “minimal rules” scenarios comes necessarily from the other steps mentioned above plus the overhead of Anubis’ additional features. If Blackwall implemented these features on top of its current version, the gaps would be smaller and could even equalize or switch! In a simplified view, a challenge scenario involves: receiving a request, evaluating it against the rules, creating a new challenge, storing it in memory, returning it to the client, receiving a solution, validating it, and issuing a cookie. The last three steps are skipped in bot flood. Because rule evaluation is mostly trivialized in this scenario, the performance gap of the “minimal rules” scenarios comes necessarily from the other steps mentioned above plus the overhead of Anubis’ additional features. If Blackwall implemented these features on top of its current version, the gaps would be smaller and could even equalize!

If I were to review and optimize blackwall for performance, I’d start by profiling the flow of the previous paragraph on both Anubis and Blackwall; compare the proportions of each step, and try to identify the bottlenecks from there.

As a closing note, this graph means the filters (Anubis and Blackwall) can keep up with 10 ms latency in 99% of the rps until reaching the limits shown in the graph. I doesn’t mean they stop working at that point, just that they start degrading.

CPU Per Request

CPU per request at the same loadCPU time per request when both servers receive the same request rate, about half of Anubis's limit with the stock rules in each scenario (400 to 1,500 req/s): the server's own code, then the kernel, hatched.BlackwallAnubisKernel time0200400600800CPU time per request (µs)Blackwall ÷ AnubisStock rulesAllowed0.42Blackwall — Stock rules, allowed, 750 req/s — 122 µs: 79 µs own code + 43 µs kernel122Anubis — Stock rules, allowed, 750 req/s — 290 µs: 245 µs own code + 44 µs kernel290Denied0.29Blackwall — Stock rules, denied, 1,500 req/s — 59 µs: 44 µs own code + 15 µs kernel59Anubis — Stock rules, denied, 1,500 req/s — 203 µs: 182 µs own code + 21 µs kernel203Bot flood0.56Blackwall — Stock rules, bot flood, 1,000 req/s — 191 µs: 177 µs own code + 15 µs kernel191Anubis — Stock rules, bot flood, 1,000 req/s — 343 µs: 313 µs own code + 30 µs kernel343Challenge solved0.46Blackwall — Stock rules, challenge solved, 500 req/s — 293 µs: 263 µs own code + 31 µs kernel293Anubis — Stock rules, challenge solved, 500 req/s — 640 µs: 563 µs own code + 77 µs kernel640Returning visitor0.44Blackwall — Stock rules, returning visitor, 750 req/s — 170 µs: 138 µs own code + 32 µs kernel170Anubis — Stock rules, returning visitor, 750 req/s — 388 µs: 338 µs own code + 50 µs kernel388Minimal rulesAllowed0.58Blackwall — Minimal rules, allowed, 750 req/s — 97 µs: 54 µs own code + 43 µs kernel97Anubis — Minimal rules, allowed, 750 req/s — 166 µs: 109 µs own code + 57 µs kernel166Denied0.39Blackwall — Minimal rules, denied, 1,500 req/s — 32 µs: 19 µs own code + 13 µs kernel32Anubis — Minimal rules, denied, 1,500 req/s — 83 µs: 64 µs own code + 18 µs kernel83Bot flood0.79Blackwall — Minimal rules, bot flood, 1,000 req/s — 165 µs: 153 µs own code + 11 µs kernel165Anubis — Minimal rules, bot flood, 1,000 req/s — 209 µs: 182 µs own code + 27 µs kernel209Challenge solved0.72Blackwall — Minimal rules, challenge solved, 500 req/s — 268 µs: 244 µs own code + 24 µs kernel268Anubis — Minimal rules, challenge solved, 500 req/s — 373 µs: 316 µs own code + 57 µs kernel373Returning visitor0.59Blackwall — Minimal rules, returning visitor, 750 req/s — 146 µs: 117 µs own code + 28 µs kernel146Anubis — Minimal rules, returning visitor, 750 req/s — 248 µs: 207 µs own code + 41 µs kernel248Large bodies, minimal rulesAllowed, 1 MiB0.75Blackwall — Large bodies, minimal rules, allowed, 1 mib, 400 req/s — 327 µs: 78 µs own code + 248 µs kernel327Anubis — Large bodies, minimal rules, allowed, 1 mib, 400 req/s — 433 µs: 135 µs own code + 299 µs kernel433Upload, 1 MiB0.76Blackwall — Large bodies, minimal rules, upload, 1 mib, 400 req/s — 343 µs: 90 µs own code + 254 µs kernel343Anubis — Large bodies, minimal rules, upload, 1 mib, 400 req/s — 453 µs: 151 µs own code + 303 µs kernel453CPU per request at the same loadCPU time per request when both servers receive the same request rate, about half of Anubis's limit with the stock rules in each scenario (400 to 1,500 req/s): the server's own code, then the kernel, hatched.BlackwallAnubisKernel time0200400600800CPU time per request (µs)Blackwall ÷ AnubisStock rulesAllowed0.42Blackwall — Stock rules, allowed, 750 req/s — 122 µs: 79 µs own code + 43 µs kernel122Anubis — Stock rules, allowed, 750 req/s — 290 µs: 245 µs own code + 44 µs kernel290Denied0.29Blackwall — Stock rules, denied, 1,500 req/s — 59 µs: 44 µs own code + 15 µs kernel59Anubis — Stock rules, denied, 1,500 req/s — 203 µs: 182 µs own code + 21 µs kernel203Bot flood0.56Blackwall — Stock rules, bot flood, 1,000 req/s — 191 µs: 177 µs own code + 15 µs kernel191Anubis — Stock rules, bot flood, 1,000 req/s — 343 µs: 313 µs own code + 30 µs kernel343Challenge solved0.46Blackwall — Stock rules, challenge solved, 500 req/s — 293 µs: 263 µs own code + 31 µs kernel293Anubis — Stock rules, challenge solved, 500 req/s — 640 µs: 563 µs own code + 77 µs kernel640Returning visitor0.44Blackwall — Stock rules, returning visitor, 750 req/s — 170 µs: 138 µs own code + 32 µs kernel170Anubis — Stock rules, returning visitor, 750 req/s — 388 µs: 338 µs own code + 50 µs kernel388Minimal rulesAllowed0.58Blackwall — Minimal rules, allowed, 750 req/s — 97 µs: 54 µs own code + 43 µs kernel97Anubis — Minimal rules, allowed, 750 req/s — 166 µs: 109 µs own code + 57 µs kernel166Denied0.39Blackwall — Minimal rules, denied, 1,500 req/s — 32 µs: 19 µs own code + 13 µs kernel32Anubis — Minimal rules, denied, 1,500 req/s — 83 µs: 64 µs own code + 18 µs kernel83Bot flood0.79Blackwall — Minimal rules, bot flood, 1,000 req/s — 165 µs: 153 µs own code + 11 µs kernel165Anubis — Minimal rules, bot flood, 1,000 req/s — 209 µs: 182 µs own code + 27 µs kernel209Challenge solved0.72Blackwall — Minimal rules, challenge solved, 500 req/s — 268 µs: 244 µs own code + 24 µs kernel268Anubis — Minimal rules, challenge solved, 500 req/s — 373 µs: 316 µs own code + 57 µs kernel373Returning visitor0.59Blackwall — Minimal rules, returning visitor, 750 req/s — 146 µs: 117 µs own code + 28 µs kernel146Anubis — Minimal rules, returning visitor, 750 req/s — 248 µs: 207 µs own code + 41 µs kernel248Large bodies, minimal rulesAllowed, 1 MiB0.75Blackwall — Large bodies, minimal rules, allowed, 1 mib, 400 req/s — 327 µs: 78 µs own code + 248 µs kernel327Anubis — Large bodies, minimal rules, allowed, 1 mib, 400 req/s — 433 µs: 135 µs own code + 299 µs kernel433Upload, 1 MiB0.76Blackwall — Large bodies, minimal rules, upload, 1 mib, 400 req/s — 343 µs: 90 µs own code + 254 µs kernel343Anubis — Large bodies, minimal rules, upload, 1 mib, 400 req/s — 453 µs: 151 µs own code + 303 µs kernel453
Share this post on:
SignalShare this post via TelegramShare this post via WhatsApp Bluesky MastodonShare this post via email
Next Post
Keeping Scrapers Out