Bonum Certa Men Certa

Good News, Bad News (and Back to Normal)

Video download link | md5sum 948cc5f102085cd52f4d356b486c2586 Major Incident and Recovery Creative Commons Attribution-No Derivative Works 4.0



Summary: When many services are reliant on the integrity of a single, very tiny MicroSD card you're only moments away from 2 days of intensive labour (recovery, investigation, migration, and further coding); we've learned our lessons and took advantage of this incident to upgrade the operating system, double the storage space, even improve the code slightly (for compatibility with newer systems)

THE good news is that Gemini is expanding faster than we predicted earlier this month. Lupa is now just 11 capsules short of 2,000 and yesterday we received some E-mails asking about Gemini downtime (we also got some inquiries over IRC, which means the Gemini capsule really matters to people).



"These things are inherently fragile; telling people to reduce the number of write operations is almost unreasonable because what good is a system you cannot use (or program) as you wish?"So why was it down? The short story is, it was a hardware failure. Not the fault of GNU/Linux or anything like that (in fact, credit to GNU/Linux for letting us fetch another complete backup of the entire system despite the whole file system being in read-only mode). There was no panic, just frustration, and based on what we heard about MicroSD-based (for boot) systems such an error was inevitable and almost predictable. The latest backup (before the "emergency" one was initiated) had been marked only a few days old (contents at most a couple of days behind).

All the services are now back online, the operating system was replaced by Debian 11, and the machine has twice as much storage space as before, which ought to permit us to do things we didn't even dare when space was tight. To reduce future downtime I also bought a spare disk (card actually) and will work on improving/reducing D-R time, as it's likely that a similar incident will happen later this year or next year. These things are inherently fragile; telling people to reduce the number of write operations is almost unreasonable because what good is a system you cannot use (or program) as you wish?

"We're hoping that tonight and tomorrow we can make up for the lost time..."Debian 11 is quite nice, but of course imperfect (perception is an impossibility). It's the first time I use Debian 11 (my wife, my sister and myself all use Debian 10 on our laptops) and maybe I'll get to write some positive things about it some time later this year (once I gain more experience/s with it).

We're hoping that tonight and tomorrow we can make up for the lost time; I hardly slept yesterday (stayed awake for about 20 hours straight, then just 4 hours of sleep) and we have a bunch of things lined up that I never managed to publish as restoring services (like IPFS and Gemini) was more pressing a task, more urgent a need.

The hardest part (to me personally) was having to go to Town for replacement components, knowing that few shops still exist (even fewer because of the pandemic) and the bigger shops are full of unmasked people who don't respect people's perimeter (it's not helping that our government likes to pretend COVID-19 is just some past event).

Recent Techrights' Posts

Slopwatch: Anti-Linux Articles Published by Bots, Dominating Google News
So a lot of the Web is Microsoft chatbot-generated anti-Linux FUD
EPO Staff Representatives Confront the President Who Says 'F--king' in Front of Female Workers Over Measurable Discrimination Against Female Colleagues
Central Staff Committee versus Lukashenko's sponsor
 
They Will Never Leave Linus Torvalds Alone, Rust is Just Another Way to Cause Instability and Infighting in Linux
We already identified the Rust "community" as troublemakers more than 5 years ago and we wrote about the evidence
Moving Away From Certificate Authorities (CAs) Like Let's Encrypt Means Taking Away From the US Government the Power to 'Censor' Sites by Revoking Certificates
Gemini capsule is cheap to run and easy (easier than a Web site) to maintain. More people disillusioned and frustrated with social control media flock to it.
BetaNews' Managing Editor Wayne William Took Charge of GNU/Linux Articles and His Articles Are Real (He Actually Wrote Them)
We are frankly relieved to see that Wayne William recognised the problem and did something about it
Links 14/02/2025: Publicity Rights Violated (ByteDance), Bribes to Trump Passed via Social Control Media 'Settlements' Again
Links for the day
Gemini Links 14/02/2025: Constitution, Cosmic DE, and More
Links for the day
Links 14/02/2025: Measles Outbreak in Texas, Zelensky Warns Russia Will Attack a NATO Country
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Thursday, February 13, 2025
IRC logs for Thursday, February 13, 2025
Gemini Links 13/02/2025: gwit and Restart
Links for the day
Links 13/02/2025: Algorithm Bots and 'Teleport' Breakthrough
Links for the day
IBM Layoffs in 'RTO' Clothing Reported by Thomas Claburn
This "hey hi" (AI) nonsense is just a go-to excuse that IBM and GAFAM (and many others) use
Still Waiting for the EU to Abolish the Illegal and Unconstitutional Court Linked to EPO Corruption and Lobbyism by the Patent Litigation Industry
Sadly, all the blogs that used to talk about those issues have been infiltrated and then completely hijacked by the very perpetrators of the illegality
Social Engineering of the Free Software Movement is a Corporate Takeover With Code of Conduct (CoC) to Drive Out or Expel Dissent
Richard Stallman (RMS) covered "cancel culture"
Links 13/02/2025: Mass Layoffs at Google (Disguised as "Buyouts"), Telecoms Price Hikes as Collusion/Price-Fixing
Links for the day
[Video] Richard Stallman Questions and Answers Session in Google's YouTube or Invidious
From last night
Gemini Links 13/02/2025: Broken Watches and Naming Types
Links for the day
Corrupt Bill Gates Worming His Way Into Richard Stallman Videos in Google's YouTube
Reputation laundering riding other people's names?
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Wednesday, February 12, 2025
IRC logs for Wednesday, February 12, 2025
Links 12/02/2025: Crytek Layoffs, Security Holes, and Giving Ukraine to Russia
Links for the day
Relaying GAFAM Talking Points and Lies Using GAFAM LLMs, or Slop Pasted in by Brittany Day
linuxsecurity.com is relaying slop, i.e. misinformation
Photos From This Evening's Talk by Dr. Richard Stallman in Torino, Maybe a Video Soon
The talk that Dr. Richard Stallman gave today (a few hours ago) was recorded and streamed
IlSoftware.it Covers Richard Stallman's Visit to Give Talks in Italy
The publication is in Italian, the talk was in English
Macho Patent Office
At the EPO there's always room for women in top roles
Gemini Links 12/02/2025: "Bream Gives Me Hiccups", Making Chinese Tea, and More
Links for the day
This is Why Codeberg Issues an Apology Today
This response was clear and relatively swift
The Register Studies (to Affirm) Reports of IBM Layoffs "at the Finance and Operations business unit"
something about that specific unit
Links 12/02/2025: SSL FUD, DEI Phase-out, Felonies Committed by MElon (Data Breaches)
Links for the day
Italian Media Covers Richard Stallman's English Talk Ahead of Tonight's Public Appearance
article in La Stampa
Destruction and Distortion of Information, Including Facts About Linux (Bonus: This is Destroying the Planet)
All that LLMs have going for them is hype, and moreover media that intentionally misrepresents them and their supposed capabilities
Google Seems to Have Just Killed All Instances of Invidious
YouTube is rapidly becoming just "another Neflix"
Microsoft Skype in a Freefall: About 20% Decrease in Site Traffic in 3 Months (Amid Microsoft Phasing Out Credits)
Microsoft axing more services/features may mean that now they scrape the bottom of the barrel and Skype will simply die, discontinuing service (like ICQ) in a matter of years
Gemini Links 12/02/2025: Depression, Gabbro, WikiTok, and More
Links for the day
Links 12/02/2025: Health, Security, and Monopolies
Links for the day
Gemini Protocol is Increasingly Important to the Net
Gemini Protocol will turn 6 this summer
Former EPO Manager Warns That the Illegal 'Court' for "Unitary Patents" Enables “Law Shopping”
Daniel X. Thomas opposed the very existence of the UPC, which any honest person could recognise was both illegal and unconstitutional
Like GAFAM, the EPO is Passing the Financial Pains to Staff
the EPO is operating illegally at this point
Morale at Microsoft Ruined by the Company Labelling Thousands of Workers 'Low Performers', Sacking Them on the Spot and Denying Them Basic Benefits
people laid off as "low performers" go to social control media to bemoan the label
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Tuesday, February 11, 2025
IRC logs for Tuesday, February 11, 2025