Bonum Certa Men Certa

Good News, Bad News (and Back to Normal)

Video download link | md5sum 948cc5f102085cd52f4d356b486c2586 Major Incident and Recovery Creative Commons Attribution-No Derivative Works 4.0



Summary: When many services are reliant on the integrity of a single, very tiny MicroSD card you're only moments away from 2 days of intensive labour (recovery, investigation, migration, and further coding); we've learned our lessons and took advantage of this incident to upgrade the operating system, double the storage space, even improve the code slightly (for compatibility with newer systems)

THE good news is that Gemini is expanding faster than we predicted earlier this month. Lupa is now just 11 capsules short of 2,000 and yesterday we received some E-mails asking about Gemini downtime (we also got some inquiries over IRC, which means the Gemini capsule really matters to people).



"These things are inherently fragile; telling people to reduce the number of write operations is almost unreasonable because what good is a system you cannot use (or program) as you wish?"So why was it down? The short story is, it was a hardware failure. Not the fault of GNU/Linux or anything like that (in fact, credit to GNU/Linux for letting us fetch another complete backup of the entire system despite the whole file system being in read-only mode). There was no panic, just frustration, and based on what we heard about MicroSD-based (for boot) systems such an error was inevitable and almost predictable. The latest backup (before the "emergency" one was initiated) had been marked only a few days old (contents at most a couple of days behind).

All the services are now back online, the operating system was replaced by Debian 11, and the machine has twice as much storage space as before, which ought to permit us to do things we didn't even dare when space was tight. To reduce future downtime I also bought a spare disk (card actually) and will work on improving/reducing D-R time, as it's likely that a similar incident will happen later this year or next year. These things are inherently fragile; telling people to reduce the number of write operations is almost unreasonable because what good is a system you cannot use (or program) as you wish?

"We're hoping that tonight and tomorrow we can make up for the lost time..."Debian 11 is quite nice, but of course imperfect (perception is an impossibility). It's the first time I use Debian 11 (my wife, my sister and myself all use Debian 10 on our laptops) and maybe I'll get to write some positive things about it some time later this year (once I gain more experience/s with it).

We're hoping that tonight and tomorrow we can make up for the lost time; I hardly slept yesterday (stayed awake for about 20 hours straight, then just 4 hours of sleep) and we have a bunch of things lined up that I never managed to publish as restoring services (like IPFS and Gemini) was more pressing a task, more urgent a need.

The hardest part (to me personally) was having to go to Town for replacement components, knowing that few shops still exist (even fewer because of the pandemic) and the bigger shops are full of unmasked people who don't respect people's perimeter (it's not helping that our government likes to pretend COVID-19 is just some past event).

Recent Techrights' Posts

The LLM Bubble is About to Implode, Gimmicks and Financial Shell Games Cannot Prevent That, Only Delay It
To inflate the bubble MElon is now doing the classic trick of buying from oneself for a fictional value
 
Gemini Links 31/03/2025: More X-Filesposting and Dreaming in Emacs
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Sunday, March 30, 2025
IRC logs for Sunday, March 30, 2025
Links 30/03/2025: Security Breaches, Crackdowns on Dissent/Rival Politicians
Links for the day
Gemini Links 30/03/2025: London Soundtrack Festival, Superbloom, gmiCAPTCHA
Links for the day
Phasing Out Vista 10 in Nations Where ~90% of Windows Users Still Rely on It
Recipe for another Microsoft disaster
The Cost of Pursuing the Much-Needed Reform/Shield Against Strategic Lawsuits Against Public Participation (SLAPPs)
“It is curious that physical courage should be so common in the world and moral courage so rare.”
Links 30/03/2025: Contagious Ideas, Signal Leak, and Squashing Lousy Patents
Links for the day
Links 30/03/2025: "Quantum Randomness" and "F-1 Visa Revoked" in US
Links for the day
Gemini Links 30/03/2025: US as a Threat, Returning to the WWW
Links for the day
Links 30/03/2025: Judge Blocks Dismantling Of VOA, Turkey Arrested Many Journalists
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Saturday, March 29, 2025
IRC logs for Saturday, March 29, 2025
Judges Would Never Rule for Men Who Strangle Women or Against Women Who Merely Wrote Articles About Abuse They Had Received From Men
We don't intend to do "trial by media", so we won't be disclosing claims and defences until it's over
Windows is an Unnatural Disaster, It is Also Avoidable
there's a wide window of opportunity opening
Gemini Links 29/03/2025: Less YouTube and More Station
Links for the day
In Some Countries, Such as Thailand, Firefox is Already Measured at Less Than 2% (One Day Firefox Will Get Blocked, Not Only Lack Support)
Web consolidation around Chrom-isms will doom the Web as we know it
Killing the News With Spam and Slop Benefits Those Whose Desire is an Uninformed Population
adoption of Free software depends indirectly on political activities/activism
Links 29/03/2025: Trademarks Battles, Fires Destroy More Than 3,000 South Korean Homes
Links for the day
Open Source Initiative (OSI) Privacy Fiasco in Detail: An Introduction
Perhaps tomorrow or perhaps next week we'll share more information about what happened and what was reported to the California Privacy Protection Agency
Links 29/03/2025: More Crackdowns on Science, "Hey Hi" Slopping is Flopping
Links for the day
IBM's BS (Bait, Switch) Regarding Ways to Stay Onboard
PIPs, RTOs, and forced relocations are just an illusion of choice (or ability to recover)
Costa Rica Almost Bankrupt Because of Microsoft
the incidents in Costa Rica are Windows incidents
Gemini Links 29/03/2025: Art of Looking, Wireguard, EMacs
Links for the day
Links 29/03/2025: Attacks on Social Security and War Updates
Links for the day
Banned evidence: Ars Technica forums censored email predicting DebConf23 death, Abraham Raji & Debian cover-up
Reprinted with permission from Daniel Pocock
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Friday, March 28, 2025
IRC logs for Friday, March 28, 2025