Bonum Certa Men Certa

Good News, Bad News (and Back to Normal)

Video download link | md5sum 948cc5f102085cd52f4d356b486c2586 Major Incident and Recovery Creative Commons Attribution-No Derivative Works 4.0



Summary: When many services are reliant on the integrity of a single, very tiny MicroSD card you're only moments away from 2 days of intensive labour (recovery, investigation, migration, and further coding); we've learned our lessons and took advantage of this incident to upgrade the operating system, double the storage space, even improve the code slightly (for compatibility with newer systems)

THE good news is that Gemini is expanding faster than we predicted earlier this month. Lupa is now just 11 capsules short of 2,000 and yesterday we received some E-mails asking about Gemini downtime (we also got some inquiries over IRC, which means the Gemini capsule really matters to people).



"These things are inherently fragile; telling people to reduce the number of write operations is almost unreasonable because what good is a system you cannot use (or program) as you wish?"So why was it down? The short story is, it was a hardware failure. Not the fault of GNU/Linux or anything like that (in fact, credit to GNU/Linux for letting us fetch another complete backup of the entire system despite the whole file system being in read-only mode). There was no panic, just frustration, and based on what we heard about MicroSD-based (for boot) systems such an error was inevitable and almost predictable. The latest backup (before the "emergency" one was initiated) had been marked only a few days old (contents at most a couple of days behind).

All the services are now back online, the operating system was replaced by Debian 11, and the machine has twice as much storage space as before, which ought to permit us to do things we didn't even dare when space was tight. To reduce future downtime I also bought a spare disk (card actually) and will work on improving/reducing D-R time, as it's likely that a similar incident will happen later this year or next year. These things are inherently fragile; telling people to reduce the number of write operations is almost unreasonable because what good is a system you cannot use (or program) as you wish?

"We're hoping that tonight and tomorrow we can make up for the lost time..."Debian 11 is quite nice, but of course imperfect (perception is an impossibility). It's the first time I use Debian 11 (my wife, my sister and myself all use Debian 10 on our laptops) and maybe I'll get to write some positive things about it some time later this year (once I gain more experience/s with it).

We're hoping that tonight and tomorrow we can make up for the lost time; I hardly slept yesterday (stayed awake for about 20 hours straight, then just 4 hours of sleep) and we have a bunch of things lined up that I never managed to publish as restoring services (like IPFS and Gemini) was more pressing a task, more urgent a need.

The hardest part (to me personally) was having to go to Town for replacement components, knowing that few shops still exist (even fewer because of the pandemic) and the bigger shops are full of unmasked people who don't respect people's perimeter (it's not helping that our government likes to pretend COVID-19 is just some past event).

Recent Techrights' Posts

Digital Sanitation Good Practices
leave behind Microsoftism
10 Days Ago Richard Stallman Gave a Long Interview in French (linuxfr.org)
English translation
Science, Not Fast Food/Junk Food
The commercial exploitation of users won't stop until users exercise full control over their software or - more broadly - their computing (including data)
[Video] Dr. Richard Stallman at Technické Univerzitě v Liberci
New/via libre-liberec.cz
Security is Desirable, But Not When the Term Security is Misused to Imply Centralisation of "Trust" (Whose?)
'Security' is not an excuse for vendor lock-in
 
What's Wrong With Liking Parrots or Birds as Pets?
They'd demonise people for speaking about freedom, no matter what they say or do
The Free Software Foundation, Which Has Appointed a 43-Year-Old President, is Looking to Add Another Board Member (or Treasurer)
expect the FSF to add more people
Richard Stallman Confirms Next Week's Talk at Technical University of Munich, We Urge EPO Staff to Attend
That's probably late enough for EPO staff to attend after work
Gemini Links 18/10/2025: Notifications and Geminaut
Links for the day
Many Red Hat People Are Leaving, But It'll Be Framed Publicly as Leaving IBM
Similarly, IBM layoffs (or "RAs" as they're called) include Red Hat layoffs
Expect More Waves of Microsoft Layoffs This Month (at Least Two Rounds Confirmed Already)
From what we can gather, assuming the recent rumours about XBox are true, there will be at least 3 waves of Microsoft layoffs this month alone
Security Issues in Cisco and Jenkins Passed Off as "Linux" Problems
Fear, Uncertainty, Doubt (FUD) tactics
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Friday, October 17, 2025
IRC logs for Friday, October 17, 2025
Slopwatch: LinuxSecurity, Linux Journal (Slashdot Media), UbuntuPIT, and Google News (Noise)
egregious plagiarism
Links 17/10/2025: Better Answers Sought After Air Crashes, "China Fans Patriotic Sentiment as Trade War With U.S. Heats Up"
Links for the day
Links 17/10/2025: Fentanylware (CheeTok) Causing Problems, Japanese Government Blasts Slop
Links for the day
The Linux Foundation Seems to Have Turned Linux.com Not Only Into a Spamfarm But Also LLM Slopfarm
it's polluting the Web, even important domains like Linux.com, with spam and LLM slop
Links 17/10/2025: UK’s Largest Breach Penalty and Windows TCO Examples
Links for the day
Go Watch Video About Librephone, Get Microsoft Ads
Very ethical company...
Campaign of Defamation Against the People Who Built NixOS (and Are Now Pushed Out From Their Own Project)
We've already grown familiar with - and resistant to - such tactics
Links 17/10/2025: Nestlé Crisis, Canada Post Versus 'Gig Economy' [sic] and Vista 11 Breaks Itself
Links for the day
Tux Machines Has Helped Separate Opinions/Analysis From News
In September 2023 we decided to split things apart and not repeat links in both sites
Tux Machines Has Improved Navigation of GNU/Linux and BSD News
Some more 'wiring' work
What a World Would Look Like If Everyone Used Free Software Only
Freedom is what matters, not "Open".
The Media Helps Microsoft, Amazon and Others (GAFAM and Beyond) Lie About Mass Layoffs Amid Valuation Bubble
The media, instead of saying that there's an "AI bubble" crashing the economy might instead choose the narrative of "jobs replaced by AI"
Bad Tempered? You Might Have Just Given Away That You're Losing the Argument
Brett Wilson LLP is fully aware that it is being investigated
Richard Stallman (RMS) is a Target of Defamation Campaigns Because of His Views on Software (But Politics Are the Excuse for Defaming Him)
Here in this site we try to refrain from politics, except in Daily Links
End of Vista 10 and Rise of GNU/Linux as Client Side Operating System
It seems certain GNU/Linux will grow in popularity over time
Taking Stock of a Week's Worth of EPO Leaks
We remain committed to exposing EPO corruption as long as it keeps happening
Mathieu Parreaux claims FINMA knew since day one
Reprinted with permission from Daniel Pocock
Calumny, Libel, Joerg Jaspert & debian-private untouchable cyberbullies
Reprinted with permission from Daniel Pocock
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Thursday, October 16, 2025
IRC logs for Thursday, October 16, 2025
Techrights Turns 19 in 3 Weeks
coverage of suppressed topics and protecting all sources/whistleblowers
International E-Waste Day Same Day as End of Vista 10
message from Akira Urushibata
The EPO's Central Staff Committee Presents Evidence That Staff Compensation Lowered While the Office Increases Income by Illegally Granting Invalid Patents
These people become millionaires by doing illegal things
Second or Third Wave of Microsoft Mass Layoffs in October 2025, This Time Portugal
Those are just the ones we know about, there may be several more
'Help Net Security' (helpnetsecurity.com) May Have Become a Slopfarm as Well
Zeljka Zorz, Editor-in-Chief at Help Net Security, was reported to us
Gemini Links 17/10/2025: Rant About Network Solutions, Strange Anomaly on Lagrange
Links for the day
EPO Staff Representation Lacks Social Dialogue With Relevant Management, Controversial and Sometimes Illegal Policies Implemented Without Necessary Input
"In this open letter, the CSC requests that the President submits an agenda item in the next available General Consultative Committee (GCC) meeting on setting up regular meetings between the CSC and the higher management of DG1."
Links 16/10/2025: Political Leftovers and Gemini Protocol Links
Links for the day
Lies Need to be Corrected
the Court never invited us
Slopwatch: Guardian Digital (linuxsecurity.com), Slashdot, Google News, and More
Maybe one day, once the bubble pops completely, Google News will just outright delist all slopfarms
Lufthansa Modern Slavery, Joerg Jaspert (ganneff) & Debian NSB Softwareentwicklung charade
Reprinted with permission from Daniel Pocock
Links 16/10/2025: US Starting More Trade Wars With China, CIA War on Venezuela
Links for the day
SUSE Blog is Still LLM Slop, Marketing Manager at SUSE Cannot Write
Would you buy from a company or seek support from a company that cannot even write (or fakes writing)?
Pretend You're Not Dead: Microsoft Spent Almost Two Decades Rebranding Things as "Cloud, Then "AI", Now "XBox" and "Quantum"
"AI" bubble pops, Microsoft harping about "quantum" already
IBM Allegedly Found New Tricks for Silent Layoffs: LPI, Then MIS (Not PIP)
Remember that "Red Hat layoffs" won't be reported after the bluewashing
Links 16/10/2025: Red Lines and Feeding of Microsoft Trolls
Links for the day
MIT as a Propaganda Mill of GAFAM, Paid by GAFAM
"the news" today
Links 16/10/2025: Lies Euphemised as ‘Dueling Versions of Reality’ and Microsoft "Open" "Hey Hi" Resorts to Porn as No Business Model Was Found
Links for the day
The Local Staff Committee Munich (Representation of the EPO's Staff) Explains When Cluster of Pregnancies May Result in Reduced Pay
"...even one week of part-time working is sufficient to reduce the salary you perceive during the entirety of your maternity leave."
Another Black Eye for 'Secure Boot', Microsoft Media Tries to Blame "Linux"
It enables Microsoft to remotely control computers, even computers that don't run Windows and never had any Microsoft software installed
Slopwatch: UbuntuPIT, linuxsecurity.com, and Various Slopfarms in Google News Attacking "Linux"
A new survey of the Web said that the majority of the Web is now slop (that's being said in the news this week)
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Wednesday, October 15, 2025
IRC logs for Wednesday, October 15, 2025
Links 16/10/2025: Increased Use of Social Control Media Surveillance in US, French Rage Over Pensions
Links for the day