Bonum Certa Men Certa

Turning Away Unwanted and/or Predatory Bots

posted by Roy Schestowitz on Sep 15, 2024

Sleep Tight

If no human will ever read it, what's the point serving?

ROGUE bots (programs without operators) are ruining the Web. One of us recently contacted Semrush Holdings, Inc. (founded by Oleg Shchegolev and Dmitri Melnikov) to complain about the misbehaving bots, which offer no benefit to anyone and basically just waste bandwidth and burn the planet. Semrush responded, but it's difficult to actually anticipate better behaviour. It's like another bubble; they probably have no concrete plan as a company (Semrush Inc. became Semrush Holdings, Inc. - one can guess why).

Companies like Semrush ruin the Web for real people. They also unnecessarily increase people's hosting bills. To them, that's just an "externality" - like LLM pests, they simply couldn't care less! Companies like these motivated us to go static; they misuse programs with a database back end (e.g. wikis) because they don't behave like people who are sane. They scrape away mercilessly and selfishly. They disregard and bypass caching or HTTP headers.

That's not to say that Gemini Protocol is free of annoying bots; we wrote about some of these before and many still traverse Geminispace for little purpose other than maintaining lists like these:

There are 4056 capsules. We successfully connected recently to 2872 of them.

Those 10,000 pages sent from Techrights were retrieved for no purpose other than Lupa gathering statistics or surveying what's out there. Since midnight today we've served 13344 requests over Gemini, yesterday it was 12917, and the day before that 14257. A high proportion of these are requests from bots.

An associate has adjusted the domain's configurations to send "429 Too Many Requests" to unwanted Web requests that might cause denial of service (at sufficiently high volume). "I think this will be an appropriate tool against bots hitting the server too hard," he said. "Changes were required in NFTables and in the Apache2 configuration," he said, and there's probably no information of use for an attacker here, as merely knowing NFTables is used barely gives an advantage.

But the very fact one needs to deploy and use NFTables means extra complexity. The misbehaving, out-of-control bots have certainly caused many sites to just throw in the towel and shut down.

Making the site and capsule serve pages fast to real visitors is of utmost importance, not gaming the numbers upwards. If you want fakes, go use Facebook; or do what Clickfraud Spamnil (Swapnil Bhartiya) does at YouTube.

In Geminispace, the capsules known to be using the Certificate Authority Let's Encrypt are a dying breed; the total has fallen again. Lupa sees only 31 such capsules today:

2576 (89.7 %) capsules are self-signed, 31 (1.1 %) use the Certificate Authority Let's Encrypt, 265 (9.2 %) are signed by another CA (may be not a trusted one).

So this coming week we might see the Certificate Authority Let's Encrypt at under 1%. It used to be in around 200 capsules or around 12% of Gemini capsules.

Other Recent Techrights' Posts

2025 Will be Fought and Fraught With LLM Slop or Fake 'Articles' (Former Media/News Sites Turning to Marketing Spam)
The elephant in the room?
Brittany Day Can Rest and Let Microsoft/Chatbots Write Fake 'Articles' About "Linux" This Christmas
Who said people don't work on Christmas? Chatbots or plagiarism-as-a-service work 24/7, every day of the year except during Microsoft downtimes
 
Microsoft Openwashing Stunts Initiative (OSI) is A Vulture in "Open" Clothing
it's quite telling that the OSI isn't protecting the Open Source Definition
Gemini Links 25/12/2024: Reality Bites and Gopher Thanks
Links for the day
Links 26/12/2024: Japan-China Mitigations and Mozambique Prison Escape (1,500 Prisoners)
Links for the day
Links 26/12/2024: Ukraine's Energy Supplies Bombed on Christmas Day, Energy Lines Cut/Disrupted in the Baltic Sea Again
Links for the day
Gemini Links 26/12/2024: Rot Economy, Self-hosted Tinylogs
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Wednesday, December 25, 2024
IRC logs for Wednesday, December 25, 2024
[Meme] Time to Also Investigate Bill Gaetz
Investigation overdue
IBM Has Almost Obliterated or Killed the Entire Fedora Community (Not IBM Staff)
Remaining Fedora insiders are well aware of this, but bringing this up (an "accusation" against IBM) might be a CoC violation
Links 25/12/2024: Fentanylware (TikTok) Scams and "Zelle Scams Lead to $870M Loss"
Links for the day
Links 25/12/2024: Windows TCO Brought to SSH, Terence Eden 'Retires'
Links for the day
Links 25/12/2024: Latest Report Front Microsoft Splinter Group, War Updates
Links for the day
Links 25/12/2024: Hong Kong Attacks Activists During Holidays, Xerox to Buy Lexmark
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Tuesday, December 24, 2024
IRC logs for Tuesday, December 24, 2024
Gemini Links 25/12/2024: Open Source Social and No Search
Links for the day
Brittany Day Connects Windows Ransomware to "Linux" Using Microsoft LLMs (FUD Galore, Zero Effort, No Accountability)
FUD and misinformation made by Microsoft LLMs again?
Links 24/12/2024: Labour Strikes and TikTok Scrambling to Prop Up Radical Politicians That Would Protect TikTok
Links for the day
Where the Population is Controlled by Skinnerboxes Inside People's Pockets (or Purses)
A very small fraction of mobile users practise or exercise freedom/control over the skinnerbox
[Meme] Coin-Operated Publishers (Gaming the Message, Buying the Narrative)
Advertise (sponsor) to 'play'
Advertisers and Their Covert Impact on Publications' Output (or Writers' Topics of Choice, as Assigned or Approved by Editors)
It cannot be trivially denied that sponsorship in the form of "advertising" impacts where publishers go (or don't go, won't go)
Terrible Year for Microsoft Windows in Cyprus
down from 86% to 72% since January
[Meme] How to Kill Unions (Staff on Shoestring Budget Cannot Afford Lawyers)
What next for the EPO? "Gig economy"?
The EPO's Staff Union (SUEPO) Takes Legal Action to Rectify the Decrease in Wages (Lessening of Purchasing Power)
here is what the union published
Gemini Links 24/12/2024: Deedum Gemini Client Gets Colour Support, Advent of Code 2024
Links for the day
Microsoft Windows Slides to New Lows in Colombia
Now Windows is at an all-time low
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Monday, December 23, 2024
IRC logs for Monday, December 23, 2024