Bonum Certa Men Certa

Turning Away Unwanted and/or Predatory Bots

posted by Roy Schestowitz on Sep 15, 2024

Sleep Tight

If no human will ever read it, what's the point serving?

ROGUE bots (programs without operators) are ruining the Web. One of us recently contacted Semrush Holdings, Inc. (founded by Oleg Shchegolev and Dmitri Melnikov) to complain about the misbehaving bots, which offer no benefit to anyone and basically just waste bandwidth and burn the planet. Semrush responded, but it's difficult to actually anticipate better behaviour. It's like another bubble; they probably have no concrete plan as a company (Semrush Inc. became Semrush Holdings, Inc. - one can guess why).

Companies like Semrush ruin the Web for real people. They also unnecessarily increase people's hosting bills. To them, that's just an "externality" - like LLM pests, they simply couldn't care less! Companies like these motivated us to go static; they misuse programs with a database back end (e.g. wikis) because they don't behave like people who are sane. They scrape away mercilessly and selfishly. They disregard and bypass caching or HTTP headers.

That's not to say that Gemini Protocol is free of annoying bots; we wrote about some of these before and many still traverse Geminispace for little purpose other than maintaining lists like these:

There are 4056 capsules. We successfully connected recently to 2872 of them.

Those 10,000 pages sent from Techrights were retrieved for no purpose other than Lupa gathering statistics or surveying what's out there. Since midnight today we've served 13344 requests over Gemini, yesterday it was 12917, and the day before that 14257. A high proportion of these are requests from bots.

An associate has adjusted the domain's configurations to send "429 Too Many Requests" to unwanted Web requests that might cause denial of service (at sufficiently high volume). "I think this will be an appropriate tool against bots hitting the server too hard," he said. "Changes were required in NFTables and in the Apache2 configuration," he said, and there's probably no information of use for an attacker here, as merely knowing NFTables is used barely gives an advantage.

But the very fact one needs to deploy and use NFTables means extra complexity. The misbehaving, out-of-control bots have certainly caused many sites to just throw in the towel and shut down.

Making the site and capsule serve pages fast to real visitors is of utmost importance, not gaming the numbers upwards. If you want fakes, go use Facebook; or do what Clickfraud Spamnil (Swapnil Bhartiya) does at YouTube.

In Geminispace, the capsules known to be using the Certificate Authority Let's Encrypt are a dying breed; the total has fallen again. Lupa sees only 31 such capsules today:

2576 (89.7 %) capsules are self-signed, 31 (1.1 %) use the Certificate Authority Let's Encrypt, 265 (9.2 %) are signed by another CA (may be not a trusted one).

So this coming week we might see the Certificate Authority Let's Encrypt at under 1%. It used to be in around 200 capsules or around 12% of Gemini capsules.

Other Recent Techrights' Posts

UEFI "Secure Boot Doesn’t Play Nice at the Moment"
UEFI "Secure Boot" does not improve security. It's an artificial obstacle in service of monopoly.
If You Want to "make your 'Windows PC' lean, mean, and fast" You Will Install GNU/Linux or Some BSD
That kind of article says a lot about IDG
 
Links 15/09/2025: Bitcoin ATMs Scam and "Conservative Cryptography" (Backdoors Fantasies)
Links for the day
EPO Imitates Microsoft: "Three Days or More Per Week" Inside the Office to Get a Desk to Work on; "the Office Breaches Its Promise Towards Staff and Acts in Breach of Its Duty of Care"
The EPO serves no actual function in Europe
Links 15/09/2025: Political Affairs, Censorship, and Copyrights
Links for the day
Gemini Links 15/09/2025: Music Genres, Invisible Networks, and Akademy 2025
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Sunday, September 14, 2025
IRC logs for Sunday, September 14, 2025
Satya's Plan B: Try to Hide the Massive Extent/Scale/Scope of Microsoft Layoffs
fewer people buy Microsoft
Red Hat News About De Facto Mass Layoffs (Bluewashing) Gone From Reddit (Censored by Gatekeepers), Still Online in The Register
With RTOs, PIPs, relocation etc. expect IBM to "shed off" many Red Hatters
Gemini Links 14/09/2025: ROOPHLOCH, Music, and Reddit
Links for the day
Slopwatch: Google News Infested With Slop (About Half of the Results for "Linux" Today)
This is the sort of junk one finds when looking for "Linux" in Google News these days
Links 14/09/2025: Ricky Hatton Dies and McDonald's Declares War on Tipping Culture
Links for the day
Links 14/09/2025: Disasters for CEOs Obsessed With Slop and Slop Companies School Like Fish
Links for the day
"Bad Shim Signature" (Microsoft 'Secure' Boot)
"Fresh install not booting"
What Microsoft Garrett and Microsoft Lunduke Have in Common
Similar tactics, different "wings"
Links 14/09/2025: US "Economy Sagging", "Michigan Economy Wobbles From Tariffs"
Links for the day
Gemini Links 14/09/2025: Minimalist Snippet Manager and Omarchy Linux
Links for the day
The Face of the Digital Far Right: Microsoft Lunduke
Microsoft Lunduke is an online extremist that belongs to and panders to the far right
20 Years Later and Academia Isn't the Same
"I never dreamed of being a professor"
'Cancel Culture' by the Right: Microsoft Lunduke Contacts People's Employers Trying to Get Them Fired
Microsoft Lunduke panders to extremists online
"Bad Shim Signature"; So 'Secure' That It Overrides Users' Preferences and Turns Itself Back on (Coercive Measure)
This was a few hours ago
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Saturday, September 13, 2025
IRC logs for Saturday, September 13, 2025
Microsoft is Rapidly Dropped From Web Servers, Shows Survey
Microsoft lost about 8% "market share" in just 3 months
Many GNU/Linux Users Report MOK (Machine Owner Key) Issues in Recent Days
many people don't report this online and never post in Reddit
We Covered UEFI 'Secure Boot' Scandals. The World Listened.
To hell with UEFI 'secure boot'
Links 13/09/2025: Escalations in East Europe and POTUS’ Health Cover-Up
Links for the day
Gemini Links 13/09/2025: Lagrange Turns 5 and Lagrange 1.19.2 Released
Links for the day
Microsoft Inside Your Linux: "Security vulnerability that allowed an attacker to bypass UEFI Secure Boot."
2 hours ago
A New Low for "Linux Journal": Promoting MICROSOFT WINDOWS Using LLM Slop
They've just jumped the shark entirely
Fake News With Fake Numbers About Microsoft
"This is what happens when the world's economy is governed by sick old men"
Slopwatch: "Google News" is Fast Becoming a Mashup of Slopfarms, Linux Journal ("LJ") is a Dump of LLM Slop
Well done, Google News. Google itself can flourish as a slopfarm mashup.
Torturing Users Who Just Want to Run GNU/Linux on Their Own PC
"Linux does not want to install"
The Register MS Still Takes Money to Hype Up "AI" in Articles by Microsoft Resellers With the Term "AI" 30+ Times in Them
Notice how many times they mention "AI"
The Apache Logo News is VERY Old, Racists and 'Anti-Woke' Bigots Look for Something to Incite Other Bigots With
Nothing to see here, move along
Linux Mint 9/11: "4th One Today..." (in Reddit)
Remember that not everyone having an issue reports it to social control media like Reddit
Nepal Will Fall Without a Single Shot Fired, Thanks to Social Control Media
Or very few shots (by the authorities)
European Corruption in the European Patent Office (EPO) Targets Culture
"In reality, the project includes a new “legal instrument” shifting administrative burden and liability on EPO staff while creating new uncertainty and externalising Amicale activities."
European Authorities, Already Bribed and Infiltrated by Microsoft, Won't Help You Find BigBlueButton, Jami, Ring, and Jitsi
Because they're paid by Microsoft and are Microsoft 'addicts' themselves
UEFI Secure Boot Failing, as Expected for Nearly 15 Years Already (Techrights Said This Since 2012)
in the media
Debian 9/11
people report this issue
Gemini and Web Links 13/09/2025: MElon's Slop Grift and "Autonomous Trains"
Links for the day
Moving From Content Management Systems (CMSs) to Static Site Generators (SSGs) Saves You Time, Makes You a Lot More Productive
try to reduce the cost (financial and computational) of running your site
Pursuing Peace Through Violence
You cannot "see" a person's mind, until the mouth opens
Leak: European Patent Office (EPO) is Now Attacking Amicale Clubs
corruption has become the norm and scientists are robbed of any dignity
Can We Please Stop Celebrating Shooters?
"An important point to hammer on is that CoCs were never intended for uniform or symmetric application"
Oracle Fraud (or Defrauding Shareholders)
"the obvious [lie] is that watts are (wasted) electricity [and] and FLOPS are computing capacity"
Geminispace is Growing Faster in 2025 Than It Did in 2024
What matters is that corporations haven't ruined it and LLM slop is extremely rare
Links 13/09/2025: China Punishes for 'Negative' Posts, US Police Unable to Find Shooter
Links for the day
Who's the Mystery Financier of SLAPP Against Techrights and Is That a Millionaire/Billionaire?
Whose idea was it to fund meritless lawsuits against my wife and I?
Slopwatch: Slow Slop Day
This distracts from or may take traffic away from the original articles, actually written by actual people
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Friday, September 12, 2025
IRC logs for Friday, September 12, 2025