It's a robot eat robot world out there

Yesterday I had the day off work and focused most of my time on working on Famous Meme Maps, I wrote an update about the choropleth changes but I also have been noticing bot traffic and spammers crawling the site. This isn't too bad on it's own, it means I'm doing something right and people are finding Famous Meme Maps, but maybe just not the people I want. There are two types of bots I'll discuss in this post, spam bots and penetration testing bots.

The first, spam bots, they're a bit more boring compared to the pen testing bots. They find a web form (like the contact form on Famous Meme Maps) and spam your form with junk mail that you usually get in your spam inbox. These are more of a nuisance than anything, as long as you don't interact with the emails or links they send they're mostly harmless. To counter these spam messages I implemented reCaptcha on the contact form to use Google's artificial intelligence solution to validate if a user is a bot or not. If someone is manually filling out the form and spamming me then this does nothing, but if they are using a script it can be hard to program around reCaptcha. So far I haven't received anymore spam messages since it's been live and I'm hoping it'll keep my inbox clean for when someone actually wants to contact me.

The second and more sinister of the two bots are ones I'm calling penetration testing bots. These bots use well formed URLs that are associated with web server configuration addresses to help determine what software is running on the server. From the logs I can see someone trying to access /test/wp-includes/wlwmanifest.xml (among other similar such URLs) on Famous Meme Maps. This corresponds to some configuration file that might be on a WordPress site. If the bots can determine there's WordPress installed and the version, there's likely a lot of zero-day exploits that can be researched to gain elevated privileges to start tinkering with the inner-workings of the website.

Luckily I don't have WordPress installed so there's nothing to find. I also don't host any personally identifying information like emails or passwords in the database, so it makes Famous Meme Maps a high-effort, low-reward scenario for any would-be hackers out there. I hope you found this update informative, I want to pull back the cover on software web development. If you want to setup your own website you'll certainly encounter these bots, and maybe many more. That's it for this morning, cheers!