Bonum Certa Men Certa

Turning Away Unwanted and/or Predatory Bots

posted by Roy Schestowitz on Sep 15, 2024

Sleep Tight

If no human will ever read it, what's the point serving?

ROGUE bots (programs without operators) are ruining the Web. One of us recently contacted Semrush Holdings, Inc. (founded by Oleg Shchegolev and Dmitri Melnikov) to complain about the misbehaving bots, which offer no benefit to anyone and basically just waste bandwidth and burn the planet. Semrush responded, but it's difficult to actually anticipate better behaviour. It's like another bubble; they probably have no concrete plan as a company (Semrush Inc. became Semrush Holdings, Inc. - one can guess why).

Companies like Semrush ruin the Web for real people. They also unnecessarily increase people's hosting bills. To them, that's just an "externality" - like LLM pests, they simply couldn't care less! Companies like these motivated us to go static; they misuse programs with a database back end (e.g. wikis) because they don't behave like people who are sane. They scrape away mercilessly and selfishly. They disregard and bypass caching or HTTP headers.

That's not to say that Gemini Protocol is free of annoying bots; we wrote about some of these before and many still traverse Geminispace for little purpose other than maintaining lists like these:

There are 4056 capsules. We successfully connected recently to 2872 of them.

Those 10,000 pages sent from Techrights were retrieved for no purpose other than Lupa gathering statistics or surveying what's out there. Since midnight today we've served 13344 requests over Gemini, yesterday it was 12917, and the day before that 14257. A high proportion of these are requests from bots.

An associate has adjusted the domain's configurations to send "429 Too Many Requests" to unwanted Web requests that might cause denial of service (at sufficiently high volume). "I think this will be an appropriate tool against bots hitting the server too hard," he said. "Changes were required in NFTables and in the Apache2 configuration," he said, and there's probably no information of use for an attacker here, as merely knowing NFTables is used barely gives an advantage.

But the very fact one needs to deploy and use NFTables means extra complexity. The misbehaving, out-of-control bots have certainly caused many sites to just throw in the towel and shut down.

Making the site and capsule serve pages fast to real visitors is of utmost importance, not gaming the numbers upwards. If you want fakes, go use Facebook; or do what Clickfraud Spamnil (Swapnil Bhartiya) does at YouTube.

In Geminispace, the capsules known to be using the Certificate Authority Let's Encrypt are a dying breed; the total has fallen again. Lupa sees only 31 such capsules today:

2576 (89.7 %) capsules are self-signed, 31 (1.1 %) use the Certificate Authority Let's Encrypt, 265 (9.2 %) are signed by another CA (may be not a trusted one).

So this coming week we might see the Certificate Authority Let's Encrypt at under 1%. It used to be in around 200 capsules or around 12% of Gemini capsules.

Other Recent Techrights' Posts

People Used to Talk
If pets can live a measurably happy life without gadgets and "apps", why can't humans?
Rust is Starting to Seem More Like Microsoft-hosted "Digital Maoism", Not a Legitimate Effort to Improve Security
Maybe this is very innocent, but they seem to have taken a solid, stable program from a high-profile Frenchman and looked for ways to marry it with GitHub, i.e. Microsoft/NSA
 
Gemini Links 08/05/2025: Practical Gemini Use Case, Shutdown of the Blanket Fort Webring
Links for the day
Links 08/05/2025: "Slop Presidency", US Government Defunds Public Broadcasting
Links for the day
Lasse Fister, Organiser of Libre Graphics Meeting, Points Out the Code of Conduct is Likely Violated by the Same People Who Promote Codes of Conduct (and Then Bully Him Into Cancelling a Keynote)
I am starting to see Lasse Fister as another victim
LLM Slop Attacks Not Only Sites of Free Software Projects But Also Bug Reporting Systems (Time-wasting, in Effect "DDoS")
Microsoft, the leading purveyor and promoter of slop, is a cancer
The Richard Stallman (RMS) "European Tour" Carries on In Spite of the Nuremberg Incident
Some people spoke about how they saw yesterday's talk
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Wednesday, May 07, 2025
IRC logs for Wednesday, May 07, 2025
The CoC Means the Founder of GNU/Linux Cannot Talk and a 72-Year-Old Man With Cancer is Somehow a "Safety" Risk?
Those who don't like RMS are not forced to attend his talks
Gemini Links 07/05/2025: A Shopping Spree and Digital Gardening
Links for the day
Links 07/05/2025: Pegasus Guilty and a Path Towards EU Without Russian Energy
Links for the day
Outsourcing GNU/Linux to Microsoft GitHub Promoted by Microsoft LLM Slop and Army Officers
Something doesn't seem right
Weaponisation of For-Profit Dockets - Part III: No More Media Lawsuits From Brett Wilson LLP This Year, One Can Only Guess Why
People leak a lot of material to Techrights because they know, based on the track record, that the sources will be protected and whatever gets published will stay online, in full, no matter how stubborn an effort (even lawsuits and blackmail) will be sent its way
Gemini Links 07/05/2025: Adopting GrapheneOS, Further Enshittification of Flickr
Links for the day
Links 07/05/2025: CISA Gutted, Debt-Saddled (Likely Insolvent) 'Open' 'AI' (Proprietary Slop) Faking Its Financial State Again
Links for the day
Finland, Lithuania, and Latvia Fortify Their Digital Border With GNU/Linux
This month's data from statCounter is particularly interesting near the Baltic Sea
The European Patent Office (EPO) Has a Very Profound Corruption Issue, Far More Urgent an Issue Than Pronouns
a rather long document
Richard Stallman Gives Public Talk at Technical University of Liberec, Czech Republic
"For programs that you could run, and for network services that could do your own computing, under what circumstances is it reasonable to trust them?"
Today We Turn 18.5
The eighteenth "and a half" anniversary
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Tuesday, May 06, 2025
IRC logs for Tuesday, May 06, 2025
Microsoft Finally Admits That XBox is ****
In this case, "enshittification" is an understatement
Another Wave of Microsoft Layoffs Comes Shortly. Microsoft Propaganda Sites and Slopforms Powered by Microsoft LLMs Already Spew Out Face-Saving Nonsense.
Based on last month's leak, some very extensive layoffs are now imminent [...] Perhaps we can expect a lot of noise, some of it spewed out by bots, to distract from or belittle the impending mass layoffs
Ubuntu Becomes Microsoft GitHub, Based on Decision Made by British Army Officer
You're hopeless, Canonical
Slopwatch: Microsoft Slop, Anti-Linux Slop, and IBM Marketing Itself as a Slop Company
Microsoft-controlled LLM spewing out garbage about "Linux"
Links 06/05/2025: Microsoft's Assassination of Skype After Years of Failure, Slop Hallucinations Are Getting Worse
Links for the day
Links 06/05/2025: Changing Places and StarGrid for PalmOS
Links for the day
Windows and Microsoft Causing Serious Data Breaches, Media Rushes to Blame That on "Linux" Somehow
While selling us some rusty old propaganda about how moving to Microsoft GitHub (Rust) will improve security
Making Site Archives More Easily Accessible (Approaching 50,000 Blog Posts)
Efforts to censor us have always backfired badly
Weaponisation of For-Profit Dockets - Part II: Hiding Behind Lawyers and Barristers Who Lack Standards so as to Engage in Classic Corporate Extortion
They're trying to scare people and they misuse their licence to operate
Links 06/05/2025: LLMs/Chatbots Attract More Scrutiny (Getting Worse Over Time), PwC Has Many Layoffs
Links for the day
Thanks for listening. How can this Morse feed be further improved?
Right now any and all feedback on the audio would be helpful
statCounter: Bing's Market Share Lower Right Now Than It Was When LLM Hype Began (With "Bing Chat")
If anybody gains at Google's expense in search, it is BRICS' alternatives such as Yandex
Gemini Links 06/05/2025: Failure and Proxmox Cluster
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Monday, May 05, 2025
IRC logs for Monday, May 05, 2025