Anyone running a publicly reachable webserver gets visitors fast. Not necessarily human ones.
This blog is produced by a static site generator and served by nginx: a handful of HTML files, some CSS, a few images, nothing else. No WordPress, no PHP, no CMS, no application behind the webserver at all - there's no login form to brute-force, no database to reach, no plugin ecosystem with known CVEs to probe for. There's essentially nothing here for a scanner to actually find - except SSH, which is reachable on the same machine, but with password authentication disabled, so a plain login guess doesn't get anyone anywhere either. For a look at who's actually visiting, I picked GoAccess - it's set up in under five minutes and gives solid basic information straight from the access log, without needing anything more elaborate. And yet it kept showing a surprising amount of traffic. Looking closer, most of it wasn't real visitors - it was automated scanning. Typical requests looked like this:
/wp-login.php
/.env
/phpmyadmin/
/cgi-bin/
/?rest_route=/batch/v1
None of these individual requests is interesting on its own. What's interesting is the sheer volume. Scanners are clearly working through large parts of the internet, just checking whether known software, configuration files, or exploitable endpoints happen to be reachable anywhere.
A static site makes suspicious traffic unusually easy to spot. If there are no PHP files on the site at all, any request for *.php is obviously bogus. The same goes for query parameters here - this site doesn't use any dynamic ones, so a request like
/?rest_route=/batch/v1
can be discarded without any further analysis. nginx can answer requests like that directly with 444, a status specific to nginx that just closes the connection without sending a normal HTTP response back at all. A lot of these requests disappear from ordinary web statistics right there. That only solves part of the problem, though - the scanner is perfectly happy to try the next path next.
The obvious next step is adding more nginx rules as new junk shows up. That works, but doesn't scale: scanners change paths, probe for new products, and recombine known attack patterns in new ways faster than anyone wants to write rules by hand.
My first actual tool for this was fail2ban, and it does solve the immediate problem - it bans IPs that misbehave. What it doesn't give much of is insight: an IP gets banned, and that's about it, without much detail on what the alert that triggered the ban actually contained. So I added CrowdSec on top instead.
CrowdSec reads nginx's logs, among other things, and recognizes typical attack patterns in them - not just "this URL path showed up," but things like "this IP produced a lot of typical probe requests in a short time." The requests nginx already recognized as scanners and answers with 444 get written into their own log:
/var/log/nginx/scanners.log
CrowdSec gets that log in addition to the regular access.log. The acquisition configuration for that can be as simple as this:
filenames:
- /var/log/nginx/*.log
labels:
type: nginx
That's enough for CrowdSec to see both ordinary requests and the scanners nginx has already sorted out. Whether the logs are actually being read and parsed can be checked with:
cscli metrics show acquisition
One thing worth knowing: CrowdSec starts reading from the end of a log file when it starts up. Existing entries aren't processed retroactively - only new requests show up in the acquisition metrics from that point on.
CrowdSec distinguishes between alerts and decisions. An alert means, roughly: CrowdSec has recognized suspicious behavior. A decision is the resulting action, for example:
ban IP 203.0.113.17 for 24h
For statistics, alerts are the more interesting of the two. The number of decisions depends entirely on how the ban rules are configured - an IP banned for a full day produces no further decisions during that time, purely as an artifact of the ban duration. Alerts describe what CrowdSec actually detected, independent of what happens with that detection afterward. Each one also comes with useful metadata: country of origin, ASN, IP address, the scenario that matched, and a timestamp.
Out of curiosity, that turned into a small world map. A Python script periodically reads CrowdSec's alerts:
cscli alerts list --since 180d --limit 0 -o json
and aggregates them by country. The map first shows the raw count of detected scanner activity per country - the more alerts from a country, the darker it's shaded. Hovering over a country additionally shows its most common CrowdSec scenarios, made a bit more readable than their technical names, things like:
HTTP probing
SSH brute force
Sensitive files
Path traversal
The first version of the map had an obvious problem: a large country simply has far more servers, clients, and compromisable systems than a small one just by virtue of its size. 500 scanner alerts from the US and 500 from Bulgaria are not the same thing. That led to a second view: alerts per million inhabitants. The script pulls the latest population figures from the World Bank API and computes
alerts / population × 1,000,000
Two different perspectives come out of that. The absolute view answers which countries actual scanner traffic mostly comes from. The relative view answers which countries produce unusually much scanner traffic relative to their population. Both numbers can drive either the table or the map's coloring.
A fairly simple question - why am I seeing so many strange requests in GoAccess? - turned into a small extra piece of infrastructure for statistics about the internet's scanner traffic. The pipeline now looks like this:
internet
|
v
nginx
|
v
obvious scanners -> scanners.log
|
v
CrowdSec
|
v
alerts
|
v
Python
|
v
aggregation by country and scenario
|
v
static HTML page with a world map
The actual website stays entirely static. CrowdSec does the detection, nginx does the first rough filtering, and a small Python script regularly turns the result into a new HTML file. Technically, none of this is a particularly big solution. That's exactly why I find it interesting: a handful of nginx rules and one security tool are enough to make visible just how much automated junk permanently lands on a completely unremarkable webserver.
Here's the resulting page itself, also reachable directly at /crowdsec.html: