Bonum Certa Men Certa

Turning Away Unwanted and/or Predatory Bots

posted by Roy Schestowitz on Sep 15, 2024

Sleep Tight

If no human will ever read it, what's the point serving?

ROGUE bots (programs without operators) are ruining the Web. One of us recently contacted Semrush Holdings, Inc. (founded by Oleg Shchegolev and Dmitri Melnikov) to complain about the misbehaving bots, which offer no benefit to anyone and basically just waste bandwidth and burn the planet. Semrush responded, but it's difficult to actually anticipate better behaviour. It's like another bubble; they probably have no concrete plan as a company (Semrush Inc. became Semrush Holdings, Inc. - one can guess why).

Companies like Semrush ruin the Web for real people. They also unnecessarily increase people's hosting bills. To them, that's just an "externality" - like LLM pests, they simply couldn't care less! Companies like these motivated us to go static; they misuse programs with a database back end (e.g. wikis) because they don't behave like people who are sane. They scrape away mercilessly and selfishly. They disregard and bypass caching or HTTP headers.

That's not to say that Gemini Protocol is free of annoying bots; we wrote about some of these before and many still traverse Geminispace for little purpose other than maintaining lists like these:

There are 4056 capsules. We successfully connected recently to 2872 of them.

Those 10,000 pages sent from Techrights were retrieved for no purpose other than Lupa gathering statistics or surveying what's out there. Since midnight today we've served 13344 requests over Gemini, yesterday it was 12917, and the day before that 14257. A high proportion of these are requests from bots.

An associate has adjusted the domain's configurations to send "429 Too Many Requests" to unwanted Web requests that might cause denial of service (at sufficiently high volume). "I think this will be an appropriate tool against bots hitting the server too hard," he said. "Changes were required in NFTables and in the Apache2 configuration," he said, and there's probably no information of use for an attacker here, as merely knowing NFTables is used barely gives an advantage.

But the very fact one needs to deploy and use NFTables means extra complexity. The misbehaving, out-of-control bots have certainly caused many sites to just throw in the towel and shut down.

Making the site and capsule serve pages fast to real visitors is of utmost importance, not gaming the numbers upwards. If you want fakes, go use Facebook; or do what Clickfraud Spamnil (Swapnil Bhartiya) does at YouTube.

In Geminispace, the capsules known to be using the Certificate Authority Let's Encrypt are a dying breed; the total has fallen again. Lupa sees only 31 such capsules today:

2576 (89.7 %) capsules are self-signed, 31 (1.1 %) use the Certificate Authority Let's Encrypt, 265 (9.2 %) are signed by another CA (may be not a trusted one).

So this coming week we might see the Certificate Authority Let's Encrypt at under 1%. It used to be in around 200 capsules or around 12% of Gemini capsules.

Other Recent Techrights' Posts

Rust People: Drain the Swap, You're Holding It Wrong
Does Rust make sense?
Slopwatch: LinuxSecurity, linuxconfig.org, and Plagiarised Phoronix
Many articles out there are nowadays fake
European Patent Office Illegally Gutting and Outsourcing Its Functions, Acting Like an Above-the-Law Commercial Business (It Won't Stop at Formalities Officers (FOs) and Classification Slop at the EPO)
breaking/violating laws and conventions
Links 19/09/2025: Lobbyist of American GAFAM Becomes Data Protection Commissioner in Europe
Links for the day
 
Links 20/09/2025: Retrocomputer, Antique Phone Experience, and More
Links for the day
Links 20/09/2025: Internet Shutdowns, Media Censorship, and Climate Worries
Links for the day
About 700 New Gemini Capsules in 13 Months (or 54 Per Month)
4.8K would represent a 20% increase
Techrights the Name Turns 15
About 6 weeks from now we turn 19
Microsoft is Running Out of Time and Floating Fake Figures, Fake Projects, Fake Narratives, Fake Excuses
Also, a lot of Microsoft's "revenue" claims are circular financing (i.e. Microsoft buying from itself, which means Ponzi-like fraud)
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Friday, September 19, 2025
IRC logs for Friday, September 19, 2025
Gemini Links 20/09/2025: Navigating the Pressures of Modern Life and SpellBinding Accidentally Wrote Another Gemini Server
Links for the day
Links 19/09/2025: Press Freedom Dying in US, Anti-Austerity Strikes in France, and Alan Rusbridger to Leave 'Prospect'
Links for the day
Offloading to the Sister Site
In the interest of not overwhelming readers
Links 19/09/2025: Coffee Club and "SpellBinding is Now Absurdly Fast"
Links for the day
Links 19/09/2025: Media Freedom Ceases to Exist in US, "Consider Dropping Twitter/X"
Links for the day
Gemini Links 19/09/2025: Thinking and Insect Bites
Links for the day
Microsoft E.E.E.: Git Will Now (or Very Soon) Fully Depend on Rust, Which is Controlled by Microsoft
Microsoft now makes Git dependent on Rust, or making Git dependent on GitHub, which is proprietary
The Right to Punch People (Apparently)
At Brett Wilson, Brett's job title is "Head of Crime" and Wilson normalises calls for violence
Slop or Fake Articles Have Turned Linux Journal From a Pioneering/Trailblazing "Linux" Magazine Into a Nuisance
some sites with former reputation - good reputation - turn into cesspools
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Thursday, September 18, 2025
IRC logs for Thursday, September 18, 2025
Brett Wilson LLP Seem to Have Had Only One Litigation Client in 2025, He Was Previously Charged, Just Like the Serial Strangler From Microsoft (Whom They Now Represent)
Karma is superstition, regulators are not
Project 2030 to Cover How "Project 2025"-Styled Anti-Media Zealots From America Targeted Techrights and Tux Machines
The common denominator is also their attacks on women
Brett Wilson LLP Failed to Meet Deadlines Set by Judge 7 Months Earlier, Tried to Ruin Our Holiday, Then Had the Audacity to Ask Us for Over 3,000 Pounds for Its Own Lateness
As a matter of principle we will never respond to assassin while we are on holiday
On Claims That After Bluewashing Red Hat Will Increasingly Become an Indian Company
Discussed this week (long and detailed)
Americans Attacking British Sites Only Months After They Leave America
We find it kind of funny if not ironic that this site, originally an American site, got legal harassment only from Americans and only months after it had moved to the UK
Despite Losing Over a Quarter Million Dollars a Year Software in the Public Interest (SPI) Gives Helping Hand to Libreboot
SPI's financial state depends a lot on its public image or its reputation
Slopwatch: Google Helps Plagiarism and Sends Traffic to Ripoff Artists
That Google as a company helps spamfarms is noteworthy
If You Want to Know the Future, Listen to the Free Software Foundation (FSF) and Andy Farnell
We're sure the FSF will have plenty of its own output
Links 18/09/2025: A Taliban Ban on Internet Access and Troubled US Job Market
Links for the day
Gemini Links 18/09/2025: Computer Literacy and Accessing Alhena's Database
Links for the day
Links 18/09/2025: US War on Media (Truth Banned, Cancel Culture by the Hard Right), NYT Chief Executive Warns Cheeto is Deploying ‘Anti-press Playbook'
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Wednesday, September 17, 2025
IRC logs for Wednesday, September 17, 2025