Bonum Certa Men Certa

Good News, Bad News (and Back to Normal)

Video download link | md5sum 948cc5f102085cd52f4d356b486c2586 Major Incident and Recovery Creative Commons Attribution-No Derivative Works 4.0



Summary: When many services are reliant on the integrity of a single, very tiny MicroSD card you're only moments away from 2 days of intensive labour (recovery, investigation, migration, and further coding); we've learned our lessons and took advantage of this incident to upgrade the operating system, double the storage space, even improve the code slightly (for compatibility with newer systems)

THE good news is that Gemini is expanding faster than we predicted earlier this month. Lupa is now just 11 capsules short of 2,000 and yesterday we received some E-mails asking about Gemini downtime (we also got some inquiries over IRC, which means the Gemini capsule really matters to people).



"These things are inherently fragile; telling people to reduce the number of write operations is almost unreasonable because what good is a system you cannot use (or program) as you wish?"So why was it down? The short story is, it was a hardware failure. Not the fault of GNU/Linux or anything like that (in fact, credit to GNU/Linux for letting us fetch another complete backup of the entire system despite the whole file system being in read-only mode). There was no panic, just frustration, and based on what we heard about MicroSD-based (for boot) systems such an error was inevitable and almost predictable. The latest backup (before the "emergency" one was initiated) had been marked only a few days old (contents at most a couple of days behind).

All the services are now back online, the operating system was replaced by Debian 11, and the machine has twice as much storage space as before, which ought to permit us to do things we didn't even dare when space was tight. To reduce future downtime I also bought a spare disk (card actually) and will work on improving/reducing D-R time, as it's likely that a similar incident will happen later this year or next year. These things are inherently fragile; telling people to reduce the number of write operations is almost unreasonable because what good is a system you cannot use (or program) as you wish?

"We're hoping that tonight and tomorrow we can make up for the lost time..."Debian 11 is quite nice, but of course imperfect (perception is an impossibility). It's the first time I use Debian 11 (my wife, my sister and myself all use Debian 10 on our laptops) and maybe I'll get to write some positive things about it some time later this year (once I gain more experience/s with it).

We're hoping that tonight and tomorrow we can make up for the lost time; I hardly slept yesterday (stayed awake for about 20 hours straight, then just 4 hours of sleep) and we have a bunch of things lined up that I never managed to publish as restoring services (like IPFS and Gemini) was more pressing a task, more urgent a need.

The hardest part (to me personally) was having to go to Town for replacement components, knowing that few shops still exist (even fewer because of the pandemic) and the bigger shops are full of unmasked people who don't respect people's perimeter (it's not helping that our government likes to pretend COVID-19 is just some past event).

Recent Techrights' Posts

Thailand: GNU/Linux Up to 6% of Desktops/Laptops, According to statCounter
Desktop Operating System Market Share Thailand
António Campinos is Still 'The Fucking President' (in His Own Words) After a Fake 'Election' in 2022 (He Bribed All the Voters to Keep His Seat)
António Campinos and the Administrative Council, whose delegates he clearly bribed with EPO budget in exchange for votes
Adrian von Bidder, homeworking & Debian unexplained deaths
Reprinted with permission from Daniel Pocock
Sainsbury’s Epic Downtime Seems to be Microsoft's Fault and Might Even Constitute a Data Breach (Legal Liability)
one of Britain's largest groceries (and beyond) chains
Matthias Kirschner, FSFE analogous to identity fraud
Reprinted with permission from Daniel Pocock
 
Comparing U.E.F.I. to B.I.O.S. (Bloat and Insecurity to K.I.S.S.)
By Sami Tikkanen
New 'Slides' From Stallman Support (stallmansupport.org) Site
"In celebration of RMS's birthday, we've been playing a bit. We extracted some quotes from the various articles, comments, letters, writings, etc. and put them in the form of a slideshow in the home page."
Suicide Cluster Cover-up tactics & Debian exposed
Reprinted with permission from Daniel Pocock
Gemini Links 19/03/2024: A Society That Lost Focus and Abandoning Social Control Media
Links for the day
Matthias Kirschner, FSFE: Plagiarism & Child labour in YH4F
Reprinted with permission from Daniel Pocock
Linux Foundation Boasting About Being Connected to Bill Gates
Examples of boasting about the association
Alexandre Oliva's Article on Monstering Cults
"I'm told an earlier draft version of this post got published elsewhere. Please consider this IMHO improved version instead."
[Meme] 'Russian' Elections in Munich (Bavaria, Germany)
fake elections
Sainsbury's to Techrights: Yes, Our Web Site Broke Down, But We Cannot Say Which Part or Why
Windows TCO?
Plagiarism: Axel Beckert (ETH Zurich) & Debian Developer list hacking
Reprinted with permission from Daniel Pocock
Links 18/03/2024: Putin Cements Power
Links for the day
Flashback 2003: Debian has always had a toxic culture
Reprinted with permission from Daniel Pocock
[Meme] You Know You're Winning the Argument When...
EPO management starts cursing at everybody (which is what's happening)
Catspaw With Attitude
The posts "they" complain about merely point out the facts about this harassment and doxing
'Clown Computing' Businesses Are Waning and the Same Will Happen to 'G.A.I.' Businesses (the 'Hey Hi' Fame)
decrease in "HEY HI" (AI) hype
Free Software Needs Watchdogs, Too
Gentle lapdogs prevent self-regulation and transparency
Gemini Links 18/03/2024: LLM Inference and Can We Survive Technology?
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Sunday, March 17, 2024
IRC logs for Sunday, March 17, 2024
Links 17/03/2024: Microsoft Windows Shoves Ads Into Third-Party Software, More Countries Explore TikTok Ban
Links for the day
Molly Russell suicide & Debian Frans Pop, Lucy Wayland, social media deaths
Reprinted with permission from Daniel Pocock
Our Plans for Spring
Later this year we turn 18 and a few months from now our IRC community turns 16
Open Invention Network (OIN) Fails to Explain If Linux is Safe From Microsoft's Software Patent Royalties (Charges)
Keith Bergelt has not replied to queries on this very important matter
RedHat.com, Brought to You by Microsoft Staff
This is totally normal, right?
USPTO Corruption: People Who Don't Use Microsoft Will Be Penalised ~$400 for Each Patent Filing
Not joking!
The Hobbyists of Mozilla, Where the CEO is a Bigger Liability Than All Liabilities Combined
the hobbyist in chief earns much more than colleagues, to say the least; the number quadrupled in a matter of years
Jim Zemlin Says Linux Foundation Should Combat Fraud Together With the Gates Foundation. Maybe They Should Start With Jim's Wife.
There's a class action lawsuit for securities fraud
Not About Linux at All!
nobody bothers with the site anymore; it's marketing, and now even Linux
Links 17/03/2024: Abuses Against Human Rights, Tesla Settlement (and Crash)
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Saturday, March 16, 2024
IRC logs for Saturday, March 16, 2024
Under Taliban, GNU/Linux Share Nearly Doubled in Afghanistan, Windows Sank From About 90% to 68.5%
Suffice to say, we're not meaning to imply Taliban is "good"
Debian aggression: woman asked about her profession
Reprinted with permission from Daniel Pocock
Gemini Links 17/03/2024: Winter Can't Hurt Us Anymore and Playstation Plus
Links for the day