Bonum Certa Men Certa

Good News, Bad News (and Back to Normal)

Video download link | md5sum 948cc5f102085cd52f4d356b486c2586 Major Incident and Recovery Creative Commons Attribution-No Derivative Works 4.0



Summary: When many services are reliant on the integrity of a single, very tiny MicroSD card you're only moments away from 2 days of intensive labour (recovery, investigation, migration, and further coding); we've learned our lessons and took advantage of this incident to upgrade the operating system, double the storage space, even improve the code slightly (for compatibility with newer systems)

THE good news is that Gemini is expanding faster than we predicted earlier this month. Lupa is now just 11 capsules short of 2,000 and yesterday we received some E-mails asking about Gemini downtime (we also got some inquiries over IRC, which means the Gemini capsule really matters to people).



"These things are inherently fragile; telling people to reduce the number of write operations is almost unreasonable because what good is a system you cannot use (or program) as you wish?"So why was it down? The short story is, it was a hardware failure. Not the fault of GNU/Linux or anything like that (in fact, credit to GNU/Linux for letting us fetch another complete backup of the entire system despite the whole file system being in read-only mode). There was no panic, just frustration, and based on what we heard about MicroSD-based (for boot) systems such an error was inevitable and almost predictable. The latest backup (before the "emergency" one was initiated) had been marked only a few days old (contents at most a couple of days behind).

All the services are now back online, the operating system was replaced by Debian 11, and the machine has twice as much storage space as before, which ought to permit us to do things we didn't even dare when space was tight. To reduce future downtime I also bought a spare disk (card actually) and will work on improving/reducing D-R time, as it's likely that a similar incident will happen later this year or next year. These things are inherently fragile; telling people to reduce the number of write operations is almost unreasonable because what good is a system you cannot use (or program) as you wish?

"We're hoping that tonight and tomorrow we can make up for the lost time..."Debian 11 is quite nice, but of course imperfect (perception is an impossibility). It's the first time I use Debian 11 (my wife, my sister and myself all use Debian 10 on our laptops) and maybe I'll get to write some positive things about it some time later this year (once I gain more experience/s with it).

We're hoping that tonight and tomorrow we can make up for the lost time; I hardly slept yesterday (stayed awake for about 20 hours straight, then just 4 hours of sleep) and we have a bunch of things lined up that I never managed to publish as restoring services (like IPFS and Gemini) was more pressing a task, more urgent a need.

The hardest part (to me personally) was having to go to Town for replacement components, knowing that few shops still exist (even fewer because of the pandemic) and the bigger shops are full of unmasked people who don't respect people's perimeter (it's not helping that our government likes to pretend COVID-19 is just some past event).

Recent Techrights' Posts

Debian Developer at Sirius Was Under the Wrong Impression That Staff Must Check or Should See E-mail All the Time (24/7 Work Attention is an Occupational Health Hazard)
My personal and professional experience with a Debian Developer (DD) at work
Techrights More Productive Than Ever Before
Today we finally crossed the 1,900-page milestone
Europe's Adoption of GNU/Linux, by Country (Now About 6%)
in Switzerland, for instance, adoption of GNU/Linux has been profoundly low
Not Only Has Adoption of Windows Vista 11 Flatlined/Plateaued, Now It is Going Down!
Did many people delete Vista 11 and install GNU/Linux instead?
 
Links 04/03/2024: Old Crisis Looming, UPC Already in Infringement of Article 6 ECHR
Links for the day
The Right to Disconnect (Meme and Very Recent References)
relatively new press
Links 04/03/2024: Techno-Babble in Tech Job Ads and Vision Pro Already Breaking Apart
Links for the day
[Meme] 'Debating' People by Subscribing Them to Lots of SPAM
Rebuttal? No, spam.
From Sexual Harassment of Women to Yet More Cybercrimes
They can be prosecuted
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Sunday, March 03, 2024
IRC logs for Sunday, March 03, 2024
Venezuela: Windows Below 70% (Laptops and Desktops), GNU/Linux Up to 7%
It's a lot higher in Cuba
ICYMI: ZDNet Financially Controlled by Microsoft
a history of censoring SJVN's Microsoft-critical articles
Argentina Joining the 4% 'Club' (GNU/Linux on Desktops and Laptops)
Data as ODF
Transparency Sets Society Free
"Convenient delusions" aren't bliss but temporary relief
[Meme] The EPO, Europe's Second-Largest Institution, Which is Contracting With Belarus
Socialist EPO
The European Patent Office's (EPO) Illegal Ban on Mass Communication Gets in the Way of Democracy
The scientific process (patents apply to science) must allow scrutiny, both from within and from the outside
Links 03/03/2024: Depression in Hong Kong, Sex 'Apps' and STIs
Links for the day
Links Gemini 03/03/2024: NixOS and NextCloud, Back Into Ricing
Links for the day
The Debian family fallacy
Reprinted with permission from Daniel Pocock
GNU/Linux Peaking in Europe, Android Measured as Higher or More Prevalent Than Windows
Android topping Windows
For Every Action There's a Reaction
Gates lobbying Modi
Like in Africa, Android Takes Control, Raking in Almost All the 'Chips' in Asia
So Microsoft has no OS majority except in Japan and Russia (and tiny Armenia).
Links 03/03/2024: Goodbye, Navalny (Funeral Reports)
Links for the day
Gemini Links 03/03/2024: A Wild Devlog Appeared and GrapheneOS Ramble
Links for the day
Gemini at 3,800+
total number of known capsules at above 3.8k
Be a Navalny
We salute Mr. Navalny
Mozilla Firefox is Back in ~2% Territories, Jeopardising Its Status as Web Browser to Test/Target/Validate With
Some new stats
[Meme] Russian Standards of Law: The Executive Branch Decides Everything
the president's kangaroo court
Up Next: The Tricky Relationship Between the Administrative Tribunal of the ILO and the European Patent Organisation (EPO)
We've moved from presidents who run a republic by consent to corrupt, unqualified, dictatorial officials who bribe for the seat (buying the votes)
IRC Proceedings: Saturday, March 02, 2024
IRC logs for Saturday, March 02, 2024
Over at Tux Machines...
GNU/Linux news for the past day
Beware Imposter Sites of Techrights (Not Techrights.com or Techrights.org)
Only trust pages accessed through the domains controlled by us
Italy visa & residence permit: Albanian Outreachy, Wikimedia & Debian tighten control over woman
Reprinted with permission from Daniel Pocock
Links 02/03/2024: Actual Journalists Under Attack, More Software Patents Being Challenged
Links for the day
Gemini Links 02/03/2024: NixOS on GPD, Meson Woes
Links for the day
statCounter March 2024 Statistics (Preliminary)
Notice Asia
Links 02/03/2024: More Lawsuits Against Microsoft, Facebook Killing Hard-To-Find News
Links for the day
ZDNet (Red Ventures) Works for Microsoft (Redmond), Many Of Its Pages Are Spam/Advertisements Paid for by Microsoft
Here is the "smoking gun"
Wikipedia Demotes CNET Due to Chatbot-Generated Spew as 'Articles'; It Should Do the Same to ZDNet (Also Red Ventures, Also Microsoft Propaganda)
Redmond Ventures?
IBM Sends Money to Microsoft
Red Hat basically helps sponsor the company that's a attacking our community
The Direction WordPress (GPL) Has Taken is an Embarrassment
it comes with strings attached
When the Cancer 'Metastasises'
We had a red flag
March in Techrights (EPO Litigation and More)
One theme we'll explore a lot when it comes to GNU/Linux is the extent to which communities truly serve communities
Don't Forget to Also Follow Tux Machines
We've split the material
Yandex Usage Has Surged Since the Invasion of Ukraine, Microsoft Fell to 0.7% (It Was 1.7% Before the 'Bing Chat' Hype Campaign)
In Soviet Russia, Bing searches user
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Friday, March 01, 2024
IRC logs for Friday, March 01, 2024
Sellout Completed: Linux Foundation Converging With the Gates Foundation
not a joke
Hitler Rants Parodies on Steve Ballmer
Parody created using clips from Downfall (Der Untergang)
With Windows This Low (27% of the "OS" Market), Steve Ballmer Would Have Thrown Another Chair
The media produced many puff pieces about Nadella at 10 (as CEO), but what has he done for Windows? Nothing.
[Meme] The Naked President
EPO Suffers From Shrinkage
Attacks on the EPC: Reality and Fiction
EPO leaks