Forum running really slow

[Gavster]

Anyone else?

Sometimes not even loading

[DeskJockey]

Yup. Couldn't get on at all this morning. Tried again just now, and it failed to load into just now (obviously).

[Matty]

It should be back up.

It was getting hammered from the 82.38.180.0/24 subnet. Some searches against that show Zenlayer, others show it as Virgin Media. If it's the latter, and you're in that range...you might be out of luck.

[Gavster]

All ticking along nicely now, thanks Matty

[mik]

That.

[DeskJockey]

It is running slowly again.

[Mito Man]

Need some cocaine for the hamsters.

[Beany]

This is what I was talking about on Facebook - Botstopper is the paid version (which we use extensively at work) and Anubis is the free version.

https://anubis.techaro.lol/docs/

The paid verison can do geoblocking using a backend called Thoth, but the free version can get a lot of the bots and block 'em. It basically uses a variety of 'sanity checks' (like getting them to do hard math - something your phone can do easily, but a cheap scraper bot doesn't have the resources to complete) and then sets a cookie for however long you want it to, to mark that connection as 'OK by me'. Anything else gets the challenge every time. It costs cheap LLM scraper farms a lot of compute time for no gain. Which I appreciate, because fuck those cunts.

Another way to help at short notice- and probably the quickest 'set and forget' hammer to swing - is to block older browser versions in apache (or your web server of choice) - a dumb LLM generated version* might look something like this:

Code: Select all

<VirtualHost *:80>
    ServerName forum.example.com
    ServerAlias www.forum.example.com
    DocumentRoot /var/www/html/phpBB3

    <Directory /var/www/html/phpBB3>
        Options -Indexes +FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>

    RewriteEngine On

    # 1. Block known AI Crawlers & Scrapers
    RewriteCond %{HTTP_USER_AGENT} (GPTBot|ChatGPT-User|ClaudeBot|Bytespider|CCBot|PerplexityBot|Diffbot|ImagesiftBot|cohere-ai) [NC,OR]

    # 2. Block Chrome / Chromium versions below 150
    RewriteCond %{HTTP_USER_AGENT} "Chrome/([1-9]|[1-9][0-9]|1[0-4][0-9])\." [NC,OR]

    # 3. Block Firefox versions below 150
    RewriteCond %{HTTP_USER_AGENT} "Firefox/([1-9]|[1-9][0-9]|1[0-4][0-9])\." [NC,OR]

    # 4. Block Safari versions below 18
    RewriteCond %{HTTP_USER_AGENT} "Version/([1-9]|1[0-7])(\.[0-9]+)* Safari" [NC]

    # Return 403 Forbidden
    RewriteRule ^ - [F,L]

    ErrorLog ${APACHE_LOG_DIR}/phpbb_error.log
    CustomLog ${APACHE_LOG_DIR}/phpbb_access.log combined
</VirtualHost>

You can tell it's dumb LLM code because it didn't even consider making it https....

The theory here is that no real person ignores the nagging to update their browser that much that they'd be more than a few versions behind (most browsers will force the issue now, updating upon opening) so it's pretty safe to be reasonably aggressive - Firefox is on 155, chrome 153, Safari 18. Bots, however, use generated UAs like Chrome 147 on Windows CE running RISC etc, and most of them aren't clever enough to use fairly modern browser agents because they're being run by the cheapest outfits the developing world can provide.

If you're worried that, say, Jobbo is running an ancient version of Netscape Navigator, which you haven't accomodated for, you can run a custom 403 page to say what browsers to use, but remember that LLM scrapers can read that, and I expect if anyones running a bot farm using agentic tools, they could feasibly adjust their methodology to suit. We've got the FB group for that sort of fallback so I'd just leave it as a basic 403.

That's a very dirty way of doing it, but I've used it as an emergency measure while I wait for a colleague to config Anubis/Botstopper for me (I'm not on the tools as much as I used to be ) and almost every time, doing that and bouncing apache brings a site back immediately.

The other option is Cloudflare, which is free to use for basic stuff. You can keep the domain with whomever and just delegate CF to do the DNS work itself, the basic proxying will catch most. You can also put in geoblocking on there with custom rules, even on the free teir - example:

Code: Select all

(http.host eq "forums.ov9.co.uk" and ip.src.continent ne "EU" and ip.src.country ne "GB")

IE "IF host is the forum, and the source contintent/country isn't EU/GB" then have it do a managed challenge (which you select later in the dialogue box)

As a security rule for the domain. Enable proxying on the domain and that'll hinder most miscreants, although as ever, the arms race continues.

My team (who manage library catalogue websites) are currently losing about 30% of our time to the latest wave of crawlers who are more advanced than they use to be - IE they're using full javascript capable browser emulators to get past basic JS challenges - previously that was considered too expensive for them, but I guess the VC money is still rolling in. It's a complete fucking nightmare.

Another tip I can give - block entire ASNs if you can - the likes of Alibaba and ChinaNet - they're huge sources of bot traffic - so is OVH, ironically for you Occasionally a 'bulletproof' host (IE one that just doesn't care about abuse mails) will come online - currently Fiber Power(AS 214483) are a major source - slinging half a million hits at one of my domains for a lightly used library that usually sees a couple of hundred hits a day. So fuck them and just 403 the entire network. Not the subnet, but the network - the Automated System - the owns the subnet.

Genuinely, forget blocking IP ranges, it's not enough - these cunts will route their traffic through domestic ISP networks in the developing world via set top boxes, because OpenAI et als subcontractors of subcontractors will pay them half a cent per IP endpoint to crawl from, and the major AI players get to play innocent and pretend they're not doing this shit themselves. You aren't expecting connections from ArgentinaNet, so block that entire ISP.

If you have a method of blocking ASNs (tricky unless your provider, er, provides it - you can do it in Cloudflare though), then use this site to help find them:
https://hackertarget.com/as-ip-lookup/

It's been a godsend for me.

Obviously a lot of this is dependant on the toolchains you use, but none of this need cost money, just a bit of time to get they head around it - which is a valuable resource. So the order I'd suggest is
1: Web Server redirects (IE 403ing old browsers) - tail -f out the web server access logs, watch them for a few minutes and look to see how many of them are coming from older browser versions, and get an LLM to write a 'fuck anything older than this off to a 403' block to just slap into the web server config. Bounce the web server service, watch the traffic stop near immediately. This works well but is a scattergun.
2: Cloudflare: Takes a bit of effort to work with, and involves working with a US company (not popular these days) but it's free for the basic tier, pretty easy to get your head around, well documented (so ChatGPT - yes, I know - can guide you how to make effective rulesets and navigate the UI) and really quite effective
3: Anubis - if you get properly pissed off and CF isn't cutting the mustard, Anubis is very, *very* powerful when you get your head around it. It's also from a non rogue state (Canada), open source and runs entirely locally, although I'd recommend using Nginx to run it, and passing that down to your actual web service as Nginx is better at handling thousands of connections at a time - although you can run it on Apache if needs be, you just need to up your mpm_<whatever>.conf to allow hundreds of connections at at time.

I'd recommend using 1 initially if the site falls over and you just want it back up, then take the time to get familiar with Cloudflare - upon reflection, Anubis really is only if you start taking this shit personally.

I realise this is a bit of an infodump - use an LLM to pull out the main points if necessary. Consider it using their own weapons against them hopefully there's some useful info in here though. I've been fighting this bollocks for a couple of years now, with varying circumstances meaning that I can use CF for some domains, some not, I don't manage some servers so I can't use anubis/botstopper, etc so the above is basically my combined knowledge on the matter.

[Beany]

Also, if you do use an LLM to sanity check/simplify this, be sure to tell it to check the current state of things on the internet - I wanged it into GeMinI pRo ExTeNdEd to make sure that despite re-reading it a couple of times, I hadn't obviously belaboured any points or inadvertantly redundantly repeated myself in a redundant fasion...(my local qwen model is broken ATM after I messed with it too much)

...and it reckoned that Anubis doesn't exist, and that browser versions are only in the mid 140s because it relied entirely on it's internal model data, not checking current info.

SkinnerLLM.jpg (122.12 KiB) Viewed 107 times

LLMs really are a bit fucking useless a lot of the time.

[Marv]

Damn, is this forum sharing resources with the aquarium in Matty's Honda-e?

Seems to much of a coincidence that since he reinstalled the aquarium, the forum has been on a go slow.

[Mito Man]

The users online numbers are a bit crazy, dead internet theory right there.

[Gavster]

Oh Christ, 7000 users online in August

[Sundayjumper]

At this precise moment it says there’s you, me & a bot. And 779 guests !

IMG_9942.jpeg (29.41 KiB) Viewed 39 times

[Beany]

Yup, the guests are crawlers. Just scraping every single thread. I'd expect someone's worked out that this place contains a large amount of high reading age english and are now using it to improve their models ability to understand various different ways of using the word 'cunt'.

The bots love library catalogue metadata too for similar reasons (minus the word cunt, mostly). I look after 200 library catalogues. Almost all of which are required to be publicly accessible, so can't just be hidden off the net only to their users.

I hate my life. But thinking more clearly, I hate OpenAI, Google and the like far more.

[Jobbo]

The users online numbers are a bit crazy, dead internet theory right there.

I read last week that 49% of the content on the internet is AI generated now. And since AI uses the internet for learning that's going to mean there's new AI stuff based on earlier AI nonsense, reinforcing it and giving it credibility.

Nice to know this forum is unlikely to be AI generated even if it's feeding them