Keeping Scrapers Out

Each visit to a website has a cost for the website host. The cost for human visitors is relatively low: we move slowly, take some time to process the contents and navigate, and mostly visit only the corners we care about. Crawlers and scrapers (terms that I use interchangeably in this article despite their different meanings), on the other hand, are automated, fast, and visit every page on the site. This is not new: robots.txt has existed for a long time.

Protections against crawling has evolved over the years, in a race with crawlers which have so far managed to keep up. I even applied to my local red team when I was taking my first steps in the industry. The age of AI has caused crawler traffic to spike, to the point where AI crawlers have been requested to identify themselves separately from other web crawlers.

Enter Anubis. Anubis “weights the soul” of a connection by issuing a proof-of-work challenge, and only letting visitors in once they completely. This is a minor annoyance to humans, who have to wait a couple of seconds for the browser to complete the challenge (which requires no input from the user at all). But those few seconds pile up for crawlers. While some bother to complete the challenge, others desist immediately or after a having to do it a few times. Here’s a simplification of how it works:

Policies allow configuring who is allowed in without challenge, who is challenged, and who is kept out regardless. The meat is in the proof-of-work though. The simplest example for the PoW is finding a nonce that, when hashed alongside some random data provided by the server for each challenge, produces a certain number of leading zeros. For example, a sample of difficulty 6 requires 6 leading zeros in the hexadecimal representation of the hash. This is the flow:

1. A challenge is issued. The server generates 32 bytes of random data.
2. The client solves the challenge by finding a nonce that produces a hash with the required number of leading zeros when combined with the random data; then sends it to the server.
3. The server validates the solution, gives the client a cookie, and marks the challenge as solved so it isn’t reused.

The proof-of-work is an embarrassingly parallel brute-force search of a nonce in which the client tries nonces until finding one that produces a correct solution. For example:

# nonce and difficulty are assumed to be unsigned.
# || represents concatenation in this example.
def solve_challenge(random_data: bytes, difficulty: int) -> int:
    nonce = 0
    while True:
        if hash(random_data || nonce).startswith('0' * difficulty):
            break
        nonce += 1
    return nonce

Server side validation on the other hand is extremely cheap, it will receive a nonce and it just needs to hash it once to verify the solution:

def verify_solution(random_data: bytes, nonce: int, difficulty: int) -> bool:
    return hash(random_data || nonce).startswith('0' * difficulty)

Anubis has seen significant adoption, even kernel.org uses it in some of its subdomains, like git.kernel.org and lore.kernel.org. I’ve also personally noticed it in personal sites, Mastodon instances, and more. It is very lightweight, has a ton of configuration options, and even white-labling for hosts that don’t want to display Anubis’ anime mascot to visitors. By default, this is what a user sees while their browser solves the challenge, which takes but a few seconds:

At some point I thought of moving my site from GitHub Pages to a Hetzner machine, and became very interested in Anubis. I ended up not doing it (hence the lack of Anubis soulweighting in this site), but still decided to try my hand at a small re-implementation. Anubis is written in Go, which in my opinion is a great call, but I felt I could squeeze out some more performance by doing it in Rust.

I’ve called it Blackwall (or b7l for short), based on the Cyberpunk system keeps AIs contained in the old internet. I built it using Tokio and Axum, which when properly utilized have the potential to beat Go’s throughput while using the same or fewer resources with decent ergonomics. I used WebAssembly to implement the challenges for four main reasons:

• I suspect many crawling libraries don’t support it yet
• I am quite familiar with writing WASM from both Rust and Go, which we commonly do in Wetware
• It lets me write the challenge logic in Rust, keeping issuance and validation logic together
• It’s a great excuse to keep tending my programming skills in an environment that makes it increasingly difficult to make the time to write manual code

Blackwall currently supports basic challenge issuance and validation, with the same configuration format as Anubis; as well as basic reverse proxying. The challenges are pure WASM, parallelized with a number of workers equal to the ceiled half of the available CPU cores. I haven’t yet benchmarked the proxying against Anubis, but I suspect Rust has some gains to offer here! You can find it here, along with a demo that sets up a simple web server with a Blackwall guard!

Here’s what it looks like:

Happy hacking!