Bonum Certa Men Certa

Turning Away Unwanted and/or Predatory Bots

posted by Roy Schestowitz on Sep 15, 2024

Sleep Tight

If no human will ever read it, what's the point serving?

ROGUE bots (programs without operators) are ruining the Web. One of us recently contacted Semrush Holdings, Inc. (founded by Oleg Shchegolev and Dmitri Melnikov) to complain about the misbehaving bots, which offer no benefit to anyone and basically just waste bandwidth and burn the planet. Semrush responded, but it's difficult to actually anticipate better behaviour. It's like another bubble; they probably have no concrete plan as a company (Semrush Inc. became Semrush Holdings, Inc. - one can guess why).

Companies like Semrush ruin the Web for real people. They also unnecessarily increase people's hosting bills. To them, that's just an "externality" - like LLM pests, they simply couldn't care less! Companies like these motivated us to go static; they misuse programs with a database back end (e.g. wikis) because they don't behave like people who are sane. They scrape away mercilessly and selfishly. They disregard and bypass caching or HTTP headers.

That's not to say that Gemini Protocol is free of annoying bots; we wrote about some of these before and many still traverse Geminispace for little purpose other than maintaining lists like these:

There are 4056 capsules. We successfully connected recently to 2872 of them.

Those 10,000 pages sent from Techrights were retrieved for no purpose other than Lupa gathering statistics or surveying what's out there. Since midnight today we've served 13344 requests over Gemini, yesterday it was 12917, and the day before that 14257. A high proportion of these are requests from bots.

An associate has adjusted the domain's configurations to send "429 Too Many Requests" to unwanted Web requests that might cause denial of service (at sufficiently high volume). "I think this will be an appropriate tool against bots hitting the server too hard," he said. "Changes were required in NFTables and in the Apache2 configuration," he said, and there's probably no information of use for an attacker here, as merely knowing NFTables is used barely gives an advantage.

But the very fact one needs to deploy and use NFTables means extra complexity. The misbehaving, out-of-control bots have certainly caused many sites to just throw in the towel and shut down.

Making the site and capsule serve pages fast to real visitors is of utmost importance, not gaming the numbers upwards. If you want fakes, go use Facebook; or do what Clickfraud Spamnil (Swapnil Bhartiya) does at YouTube.

In Geminispace, the capsules known to be using the Certificate Authority Let's Encrypt are a dying breed; the total has fallen again. Lupa sees only 31 such capsules today:

2576 (89.7 %) capsules are self-signed, 31 (1.1 %) use the Certificate Authority Let's Encrypt, 265 (9.2 %) are signed by another CA (may be not a trusted one).

So this coming week we might see the Certificate Authority Let's Encrypt at under 1%. It used to be in around 200 capsules or around 12% of Gemini capsules.

Other Recent Techrights' Posts

Revisiting Julian Assange's Excellent Talk, His First Talk Since 2019 (Tactful and Almost Invulnerable to 'Cheap Shots')
Assange need not be politically-correct or self-censor
Mozilla is GAFAM, HTTPS is Monopolies
Firefox used to boast that it would make the Web more accessible. Today's Mozilla is rowing in the opposite direction.
 
One Step Closer to the End of Microsoft's XBox
XBox sales are down over 50% in the past year
GNU/Linux Flaring Up in ASEAN
We said we'd not post statCounter for a few months
Gemini Links 04/10/2024: Asteroid City and Retro Gaming
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Thursday, October 03, 2024
IRC logs for Thursday, October 03, 2024
Resting Time
we deserve a short break - even if only for tomorrow
Wikileaks Revelations About the History of IBM and Its Role in the Cold War
IBM is still an ICBM company (to this very date)
Windows Kills More Than Most Wars (But the Media Casually Ignores the Death Toll of Microsoft)
The bottom line is, many people are dying, they die due to Microsoft, and the media fails us by not informing us and failing to even name the principal culprit
Gemini Links 03/10/2024: RetroChallenge and Change of Online Habits
Links for the day
Links 03/10/2024: Quantum Computer Vapourware (as Usual) and Samsung Layoffs
Links for the day
Links 03/10/2024: "Hey Hi" Scandals and Copyright/Trademark Disputes
Links for the day
Invidious Seems to be Nearing 'End of Life' After Repeated Crackdowns by Google/Alphabet/YouTube
To Free software users, YouTube ought to become a "no-no"
Links 03/10/2024: Climate Issues and Tensions in East Asia
Links for the day
Like a Marketing Department of Microsoft, Canonical Sells Back Doors and Surveillance as "Confidential" and "Hey Hi" (AI)
Notice how Canonical has made no statement critical of Microsoft for years
Gemini Links 03/10/2024: Frozen Tofu and SGI O2
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Wednesday, October 02, 2024
IRC logs for Wednesday, October 02, 2024
Links 02/10/2024: Microsoft Spying on Windows Users Grows, Microsoft's Surveillance Arm LinkedIn Used to Highlight Employment Crisis
Links for the day
Links 02/10/2024: Students Who Can’t Read Books and Dead Butt Syndrome
Links for the day
Gemini Links 02/10/2024: GNU/Linux Distros, Flat-File Databases, and How the Web ate Gopher
Links for the day
Technology: rights or responsibilities? - Part II
By Dr. Andy Farnell
A Cost-Free Bribe From Microsoft
Daniel Stenberg is not dumb, but he seems rather gullible or unprincipled
Plans for the Site's 19th Year
Like TechDirt, we expect to devote more efforts/time to covering free speech online
Network Getting Faster
Loading up the site in 0.077 seconds
The Manchester Experience
Yesterday Tux Machines served 436,897 Web hits
If Red Hat Has Mass Layoffs This Year, Nobody Will Tell You About It
We seem to have entered a strange quasi-cosmic era wherein layoffs aren't disclosed anymore and news sites don't bother to report them, either
IBM, Kyndryl, Subsidiaries (Like Red Hat) and Silent Layoffs
Kyndryl follows in IBM's footsteps with rolling layoffs likely affecting thousands
Anniversaries and New Beginnings
The world needs more transparency and far less secrecy
Links 02/10/2024: Microsoft Kills Off HoloLens, Media Discusses Assange Speech
Links for the day
Gemini Links 02/10/2024: New Car, Broadband, and Gemtexter 3.0.0
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Tuesday, October 01, 2024
IRC logs for Tuesday, October 01, 2024