Bonum Certa Men Certa

Turning Away Unwanted and/or Predatory Bots

posted by Roy Schestowitz on Sep 15, 2024

Sleep Tight

If no human will ever read it, what's the point serving?

ROGUE bots (programs without operators) are ruining the Web. One of us recently contacted Semrush Holdings, Inc. (founded by Oleg Shchegolev and Dmitri Melnikov) to complain about the misbehaving bots, which offer no benefit to anyone and basically just waste bandwidth and burn the planet. Semrush responded, but it's difficult to actually anticipate better behaviour. It's like another bubble; they probably have no concrete plan as a company (Semrush Inc. became Semrush Holdings, Inc. - one can guess why).

Companies like Semrush ruin the Web for real people. They also unnecessarily increase people's hosting bills. To them, that's just an "externality" - like LLM pests, they simply couldn't care less! Companies like these motivated us to go static; they misuse programs with a database back end (e.g. wikis) because they don't behave like people who are sane. They scrape away mercilessly and selfishly. They disregard and bypass caching or HTTP headers.

That's not to say that Gemini Protocol is free of annoying bots; we wrote about some of these before and many still traverse Geminispace for little purpose other than maintaining lists like these:

There are 4056 capsules. We successfully connected recently to 2872 of them.

Those 10,000 pages sent from Techrights were retrieved for no purpose other than Lupa gathering statistics or surveying what's out there. Since midnight today we've served 13344 requests over Gemini, yesterday it was 12917, and the day before that 14257. A high proportion of these are requests from bots.

An associate has adjusted the domain's configurations to send "429 Too Many Requests" to unwanted Web requests that might cause denial of service (at sufficiently high volume). "I think this will be an appropriate tool against bots hitting the server too hard," he said. "Changes were required in NFTables and in the Apache2 configuration," he said, and there's probably no information of use for an attacker here, as merely knowing NFTables is used barely gives an advantage.

But the very fact one needs to deploy and use NFTables means extra complexity. The misbehaving, out-of-control bots have certainly caused many sites to just throw in the towel and shut down.

Making the site and capsule serve pages fast to real visitors is of utmost importance, not gaming the numbers upwards. If you want fakes, go use Facebook; or do what Clickfraud Spamnil (Swapnil Bhartiya) does at YouTube.

In Geminispace, the capsules known to be using the Certificate Authority Let's Encrypt are a dying breed; the total has fallen again. Lupa sees only 31 such capsules today:

2576 (89.7 %) capsules are self-signed, 31 (1.1 %) use the Certificate Authority Let's Encrypt, 265 (9.2 %) are signed by another CA (may be not a trusted one).

So this coming week we might see the Certificate Authority Let's Encrypt at under 1%. It used to be in around 200 capsules or around 12% of Gemini capsules.

Other Recent Techrights' Posts

Bailing Out GAFAM, Giving Taxpayers' Money to Failing Companies, and Trying to Outlaw Lawsuits Against Them
What would the late Lincoln have said?
 
Slopwatch: Slopfarms All Over Google News and Real News Sites Pushed Out of Visibility
Google News is dying (as a tool of value)
Gemini Links 25/08/2025: Numeric-only VM and Alhena 5.3.0
Links for the day
Links 25/08/2025: ‘Panama Playlists’ and Live Nation/Ticketmaster Suit Aims at Class Action
Links for the day
Gemini Links 25/08/2025: Empathy Towards Autistic People and Old Gadgets
Links for the day
Links 25/08/2025: Datacentres Versus Water Supplies and "The IPv6 Divide"
Links for the day
Links 25/08/2025: Data Breaches, Politics, and Financial Strain
Links for the day
GNU/Linux Distros Ought to Replace Firefox (and Firefox ESR) With Something Like LibreWolf
Perhaps it's come to replace Firefox
Father of Julian Assange Said the US Government Was Trying to Bankrupt WikiLeaks, Now the Assange Family Promotes Fake Currencies
Using the name for bad purposes?
Software Freedom Conservancy (SFC) Inc. Lost 2 Million Dollars Last Year and Its Chief Took a Salary Increase of Almost $6,000
Another year or two like this... and the SFC will be bankrupt [...] Hallmark of mismanagement
The "New Techrights" Turns Two Very Soon
Accomplishing something each year is what's important, not merely "finishing" another year
Gulf Nations Leave Microsoft Behind
How much lower will Microsoft stoop in an effort to raise money from oil-rich lenders?
How to Combat IRC Trolls (in Our Experience)
Today I want to share my experience (or knowledge) of how to deal with IRC trolls
The Register MS Needs to Stop Participating in the "Hey Hi" (AI) Hype, But It Gets Paid to Participate in This Hype
the publisher (The Register MS) wants to have it both ways
Gemini Links 24/08/2025: Living With Your Parents, Zürich Zoo, and Macondo
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Sunday, August 24, 2025
IRC logs for Sunday, August 24, 2025
Gemini Links 24/08/2025: Signal on OpenBSD and Keyboard Layouts Compared
Links for the day
Men Who Abuse Women Should Never Spend Over 3 Years of the UK High Court's Time
This demonstrates that we need a reform in the UK
Links 24/08/2025: Microsoft Settles Data Breach Lawsuits and Climate Change Causes Heatwaves, Water Shortages
Links for the day
CachyOS is Rising Fast, But Slopfarms Are 'Googlebombing' It
CachyOS receives more media attention
No Reason for Red Hat Relief Yet (Layoff Rumours)
the execution could be stalled, delayed, or scheduled for some time after people come back from holiday
GNU/Linux 6%, Windows 60% in Venezuela, Suggests statCounter
The cash cows are dying
Mass Layoffs Continue at Microsoft This Month (Remaining Workers See Conditions That Deteriorate)
So far this month (one week remaining) we saw at least two waves of layoffs at Microsoft
How SPAM E-mails With Windows-Centric Files Get Twisted as Linux Threats, Then Slopfarms Spread the Word
Fear, Uncertainty, Doubt/Fear-mongering/Dramatisation
Links 24/08/2025: Heatwaves Threaten Workers, Maldives Versus Press freedom
Links for the day
Gemini Links 24/08/2025: Digital Cameras and Printers
Links for the day
Links 24/08/2025: GAFAM Lie About Pollution and Slop's Carbon Footprint, The Guardian Says Slop ("Hey Hi") is a Bubble That Will Send Stock Markets Into a Freefall
Links for the day
80% of the Sponsored (Fake) Articles in The Register MS Are Promotions of Ponzi Schemes (Unethical Money), the Rest is Banned Chinese Business
Is that an ethical way to make money? No.
The UEFI Restricted Boot 'Time Bomb' is About to Go Off in a Few Weeks
Garrett was the first person to face sanctions (like muting) in our IRC channels because of his abuse; worse yet, he hijacked other people's names and then locked them out of their own accounts
Should Currys PCWorld Start Voiding Warranties of Users of Vista 11?
If a person's laptop has a mechanical issue, should this person replace GNU/Linux with Vista 11 for the repair shop? Only to damage the SSD?
Newer is Not Always Better, and It's Possible That 'Peak' is the Past
People creating their own platforms means progress, whereas centralisation (like moving from blogs to social control media) is the opposite of progress
LLM Hype is Sowing Destruction: It Contributes to DDoS Attacks and Makes the Web Less Accessible (JavaScript "R U Human?" Tests)
If it was googlebot, it would be possible to argue that you'd at least then get referral traffic from Google Search. With LLMs, all you get is plagiarised.
Links 24/08/2025: New York Times Talks About Hey Hi (AI) Bubble
Links for the day
Gemini Links 24/08/2025: Upgrading Debian and Mobile-indifferent Design
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Saturday, August 23, 2025
IRC logs for Saturday, August 23, 2025