Bonum Certa Men Certa

Turning Away Unwanted and/or Predatory Bots

posted by Roy Schestowitz on Sep 15, 2024

Sleep Tight

If no human will ever read it, what's the point serving?

ROGUE bots (programs without operators) are ruining the Web. One of us recently contacted Semrush Holdings, Inc. (founded by Oleg Shchegolev and Dmitri Melnikov) to complain about the misbehaving bots, which offer no benefit to anyone and basically just waste bandwidth and burn the planet. Semrush responded, but it's difficult to actually anticipate better behaviour. It's like another bubble; they probably have no concrete plan as a company (Semrush Inc. became Semrush Holdings, Inc. - one can guess why).

Companies like Semrush ruin the Web for real people. They also unnecessarily increase people's hosting bills. To them, that's just an "externality" - like LLM pests, they simply couldn't care less! Companies like these motivated us to go static; they misuse programs with a database back end (e.g. wikis) because they don't behave like people who are sane. They scrape away mercilessly and selfishly. They disregard and bypass caching or HTTP headers.

That's not to say that Gemini Protocol is free of annoying bots; we wrote about some of these before and many still traverse Geminispace for little purpose other than maintaining lists like these:

There are 4056 capsules. We successfully connected recently to 2872 of them.

Those 10,000 pages sent from Techrights were retrieved for no purpose other than Lupa gathering statistics or surveying what's out there. Since midnight today we've served 13344 requests over Gemini, yesterday it was 12917, and the day before that 14257. A high proportion of these are requests from bots.

An associate has adjusted the domain's configurations to send "429 Too Many Requests" to unwanted Web requests that might cause denial of service (at sufficiently high volume). "I think this will be an appropriate tool against bots hitting the server too hard," he said. "Changes were required in NFTables and in the Apache2 configuration," he said, and there's probably no information of use for an attacker here, as merely knowing NFTables is used barely gives an advantage.

But the very fact one needs to deploy and use NFTables means extra complexity. The misbehaving, out-of-control bots have certainly caused many sites to just throw in the towel and shut down.

Making the site and capsule serve pages fast to real visitors is of utmost importance, not gaming the numbers upwards. If you want fakes, go use Facebook; or do what Clickfraud Spamnil (Swapnil Bhartiya) does at YouTube.

In Geminispace, the capsules known to be using the Certificate Authority Let's Encrypt are a dying breed; the total has fallen again. Lupa sees only 31 such capsules today:

2576 (89.7 %) capsules are self-signed, 31 (1.1 %) use the Certificate Authority Let's Encrypt, 265 (9.2 %) are signed by another CA (may be not a trusted one).

So this coming week we might see the Certificate Authority Let's Encrypt at under 1%. It used to be in around 200 capsules or around 12% of Gemini capsules.

Other Recent Techrights' Posts

Tesla's Debt More Than Doubled in 2 Years and the Company Will Operate in the Red (at a Loss) Quite Soon
If your first-quarter net income is $409 million and you borrow billions from banks, plus interest to pay on those loans, then you're not far from returning to losses
Microsoft and Windows Have Many Back Doors, But LLM Slop Keep Claiming That Linux Has "Backdoor"
It's another example of LLM slop as FUD amplifier, via slopfarms as well
 
Rumours of Mass Layoffs at Red Hat Next Week (August 11th, 2025)
The eleventh means next Monday
IBM is Shutting Down (Piecewise)
IBM is basically being liquidated
The Debian Language Police Department (PD)
"there has never been complaints about anyone that was offended by this -off package"
When The Register MS Says "Linux Backdoor" It Actually Talks About Malware
The leading story in The Register US/MS this morning is Microsoft
Microsoft Windows Fell to 19% "Market Share" in Montenegro
Microsoft must be well aware of this trend
Why We Also Include Gopher Links in Our Gemini (Protocol) Links
There are still many people who use Gopher to relay their messages (like blog posts). They're mostly technical people.
Shouting is an Indication of a Lack of Convincing Argument
Beware what they are attempting to distract from
Mongolia: Microsoft Windows at All-Time Low
in 2009 when Windows was at 99.45% in Mongolia the company was "worth" less than 200 billion dollars
About a Quarter of Today's "linux" News in Google News Came From One Domain and It's a Slopfarm
Not kidding!
Gemini Links 05/08/2025: Zombie Threat and Switching to NixOS
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Monday, August 04, 2025
IRC logs for Monday, August 04, 2025
ChatGPT in Trouble
Watch out for the newer buzzwords
The Register MS Links to the Wrong statCounter Page
They link to older data
Dr. Andy Farnell Explains How Google Turned From "Librarian" Into "Oracle", Telling Us What to Think Instead of Where to Look
Google was always a lousy librarian
Microsoft Layoffs Continue in August 2025
If Microsoft is doing so well, how come about 10 rounds of layoffs in about 7 months in 2025?
In Many Countries Vista 11 Adoption Stalled or Became Negative
Not just because people move to GNU/Linux
Microsofters' Lawyers Are Name-calling and Insulting Microsoft Critics, Even Their Spouses
How not to win arguments
Flagging or Tagging Slop That We Find Online
Right now we use ImageMagick
Links 04/08/2025: Very Bad Weather and Travel Restrictions in China
Links for the day
Gemini Links 04/08/2025: Misiamisia and Mobile Linux
Links for the day
Microsoft's Stock is Like a Religion, Microsoft Goes Into 'Hiding' (From Shareholders)
like a religious person or devout believer, the media just parrot anything Microsoft says
Links 04/08/2025: 80 Years Since Last Nuclear War, IPv6 in China
Links for the day
Groklaw Static Site Relaunches With New Theme, But Many Pages and All the Comments Are Missing
We suppose that's still a lot better than the site being offline, as it was for several months
"For Five decades; For freedoms; For all users" (Original EMACS Turns 50 Next Year)
Linus Benedict Torvalds was only 6 when EMACS started
In Spain, Microsoft's Search Engine Market Share Fell to 2%
16 years have passed since Bing was introduced
Protecting GNU/Linux-Centric Journalism From Serial Sloppers
Unoriginal slop is taking away traffic from the people who did all the real work
It Looks Like Managers at Oracle Now Use LLM Slop to Write Blog Posts
Did he cheat by prompting LLMs for mindless text "filler"?
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Sunday, August 03, 2025
IRC logs for Sunday, August 03, 2025
Gemini Links 04/08/2025: Qubes OS and Curious crypto case of certificates (CCCC)
Links for the day
They Tell Us That "Cloud Storage" is Safe and Robust to Incidents Like Fires
Do you have backups? Where are they and who controls them?
"Allowing SDL to default to Wayland caused a number of customer issues so keep the default at X11 for now"
2025 is another year of Wayland ambitions. It's also a year of self-fulfilling prophecies.
In The United Kingdom (UK), Microsoft Search (Bing) Falls to All-Time Low
Grow? What grow??? It's collapsing.
GNU/Linux Reaches 5% in Oman
Some GNU/Linux distros are made in Oman
Google's "AI Mode" is a Pathetic Joke Prematurely Introduced in the UK (Like "Bard", Which Sank the Company's Shares)
what Google "thinks" about PCLinuxOS
What the Free Software Foundation Started Four Decades Ago is Becoming Mainstream
"Four decades; Four freedoms; For all users"
Doing a Better Job at Labelling Slop Images
we'll label screenshots that contain slop, typically with red-coloured text overlay
Social Control Media is Out of Style
What's your excuse for wasting time on (or in) it?
Maldives: GNU/Linux at All-Time High, Windows at New Lows
data from statCounter shows a reassuring trend
Efficiency is Good, So Why Won't Governments Cull LLM Companies Using Stronger, Stringent Policies?
Like every bubble that ever existed, including some recent ones, an end will come
The Defunct Site LinuxConfig Has Published a Fake Article About Richard Stallman Using LLM Slop, Which Stallman Calls "Bullshit Generator"
Worse yet, it is writing using a "Bullshit Generator" (the term used by Stallman) about Stallman's health
Microsoft Windows Falls to All-Time Lows in Morocco and Algeria
About 70% or even less
StopGenAI in the Cyber Show (C|S)
covering a theme that we too covered a lot lately
Gemini Links 03/08/2025: Once-a-Decade Couch Shopping and Blessings in Disguise
Links for the day
Links 03/08/2025: Political Catch-up, Global Warming, and Hunger
Links for the day
Brittany Day Entered LLM Slop Into LinuxSecurity.com and Something Hilarious Happened: The Site is "Exploited"
The brainless, effortless copypasta of "slop artists" shows its limits
Links 03/08/2025: Microsoft Exchange 0-day Exploited and Avoidable Nuclear Escalation
Links for the day
Next Month 'New Techrights' Turns Two
Next month, on the fourth week, it'll be 2 years since the migration
Definitely Not a Ponzi Scheme
Bitcoin v Microsoft
Online Safety Act Tries to Accomplish the Impossible
All I can say is, "good luck with that!"
The Electronic Frontier Foundation (EFF) is a Billionaires' Lobby
Billionaires that control tech companies
Microsoft Borrows 3 Billion Dollars Per Month, a Company Truly Worth Trillions Would Not Do This
if Windows (and Office) "market share" fell from about 90% to barely 30%, how come Microsoft is now "valued" at 20 times more?
It's Even Worse Than Microsoft Lunduke Puts It; GNOME is SLAPPing Journalists
In our experience, GNOME is so malicious - some elements of it in particular - that it would launch multiple simultaneous SLAPP campaigns not only against journalists but also their spouses
GNU/Linux Adoption Reaches All-Time Highs in Chile, statCounter Indicates
This month marks 4 years since Vista 11 came out (as a fake "leak") and some surveys still measure its adoption at less than 40%
Slop Will Not Change the World
Some of us grow up sooner and leave that nonsense behind (or altogether avoid/skip it)
Gemini Links 03/08/2025: Nostalgia and TOFU
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Saturday, August 02, 2025
IRC logs for Saturday, August 02, 2025