Bonum Certa Men Certa

Good News, Bad News (and Back to Normal)

Video download link | md5sum 948cc5f102085cd52f4d356b486c2586 Major Incident and Recovery Creative Commons Attribution-No Derivative Works 4.0



Summary: When many services are reliant on the integrity of a single, very tiny MicroSD card you're only moments away from 2 days of intensive labour (recovery, investigation, migration, and further coding); we've learned our lessons and took advantage of this incident to upgrade the operating system, double the storage space, even improve the code slightly (for compatibility with newer systems)

THE good news is that Gemini is expanding faster than we predicted earlier this month. Lupa is now just 11 capsules short of 2,000 and yesterday we received some E-mails asking about Gemini downtime (we also got some inquiries over IRC, which means the Gemini capsule really matters to people).



"These things are inherently fragile; telling people to reduce the number of write operations is almost unreasonable because what good is a system you cannot use (or program) as you wish?"So why was it down? The short story is, it was a hardware failure. Not the fault of GNU/Linux or anything like that (in fact, credit to GNU/Linux for letting us fetch another complete backup of the entire system despite the whole file system being in read-only mode). There was no panic, just frustration, and based on what we heard about MicroSD-based (for boot) systems such an error was inevitable and almost predictable. The latest backup (before the "emergency" one was initiated) had been marked only a few days old (contents at most a couple of days behind).

All the services are now back online, the operating system was replaced by Debian 11, and the machine has twice as much storage space as before, which ought to permit us to do things we didn't even dare when space was tight. To reduce future downtime I also bought a spare disk (card actually) and will work on improving/reducing D-R time, as it's likely that a similar incident will happen later this year or next year. These things are inherently fragile; telling people to reduce the number of write operations is almost unreasonable because what good is a system you cannot use (or program) as you wish?

"We're hoping that tonight and tomorrow we can make up for the lost time..."Debian 11 is quite nice, but of course imperfect (perception is an impossibility). It's the first time I use Debian 11 (my wife, my sister and myself all use Debian 10 on our laptops) and maybe I'll get to write some positive things about it some time later this year (once I gain more experience/s with it).

We're hoping that tonight and tomorrow we can make up for the lost time; I hardly slept yesterday (stayed awake for about 20 hours straight, then just 4 hours of sleep) and we have a bunch of things lined up that I never managed to publish as restoring services (like IPFS and Gemini) was more pressing a task, more urgent a need.

The hardest part (to me personally) was having to go to Town for replacement components, knowing that few shops still exist (even fewer because of the pandemic) and the bigger shops are full of unmasked people who don't respect people's perimeter (it's not helping that our government likes to pretend COVID-19 is just some past event).

Recent Techrights' Posts

Tentative Summary of Things to Publish in Project 2030
I'll still be in my forties by then
Rust People: Drain the Swap, You're Holding It Wrong
Does Rust make sense?
Slopwatch: LinuxSecurity, linuxconfig.org, and Plagiarised Phoronix
Many articles out there are nowadays fake
 
Climate Breakdown Means We'll be Publishing More, Not Less
Press freedom will be a common, recurring theme
Our 5-Year Geminispace Anniversary is Coming Up
I still remember when Gemini Protocol was quite new
It's Right to Point Out Violence From the Right
Violence is a recurring theme
Web Browsers That "Do Hey Hi" (AI)
State-of-the-art plagiarism or "autocomplete on steroids" (not coined by us, nevertheless a nice description) don't have much/any prospect
Links 20/09/2025: Hardware Projects in View, Some Independent Publishers About Russia Prosper After Cheeto Cuts Funding
Links for the day
Gemini Links 20/09/2025: Options and TV Time Machine
Links for the day
Links 20/09/2025: Retrocomputer, Antique Phone Experience, and More
Links for the day
Links 20/09/2025: Internet Shutdowns, Media Censorship, and Climate Worries
Links for the day
About 700 New Gemini Capsules in 13 Months (or 54 Per Month)
4.8K would represent a 20% increase
Techrights the Name Turns 15
About 6 weeks from now we turn 19
Microsoft is Running Out of Time and Floating Fake Figures, Fake Projects, Fake Narratives, Fake Excuses
Also, a lot of Microsoft's "revenue" claims are circular financing (i.e. Microsoft buying from itself, which means Ponzi-like fraud)
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Friday, September 19, 2025
IRC logs for Friday, September 19, 2025
Gemini Links 20/09/2025: Navigating the Pressures of Modern Life and SpellBinding Accidentally Wrote Another Gemini Server
Links for the day
Links 19/09/2025: Press Freedom Dying in US, Anti-Austerity Strikes in France, and Alan Rusbridger to Leave 'Prospect'
Links for the day
European Patent Office Illegally Gutting and Outsourcing Its Functions, Acting Like an Above-the-Law Commercial Business (It Won't Stop at Formalities Officers (FOs) and Classification Slop at the EPO)
breaking/violating laws and conventions
Offloading to the Sister Site
In the interest of not overwhelming readers
Links 19/09/2025: Coffee Club and "SpellBinding is Now Absurdly Fast"
Links for the day
Links 19/09/2025: Lobbyist of American GAFAM Becomes Data Protection Commissioner in Europe
Links for the day
Links 19/09/2025: Media Freedom Ceases to Exist in US, "Consider Dropping Twitter/X"
Links for the day
Gemini Links 19/09/2025: Thinking and Insect Bites
Links for the day
Microsoft E.E.E.: Git Will Now (or Very Soon) Fully Depend on Rust, Which is Controlled by Microsoft
Microsoft now makes Git dependent on Rust, or making Git dependent on GitHub, which is proprietary
The Right to Punch People (Apparently)
At Brett Wilson, Brett's job title is "Head of Crime" and Wilson normalises calls for violence
Slop or Fake Articles Have Turned Linux Journal From a Pioneering/Trailblazing "Linux" Magazine Into a Nuisance
some sites with former reputation - good reputation - turn into cesspools
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Thursday, September 18, 2025
IRC logs for Thursday, September 18, 2025
Brett Wilson LLP Seem to Have Had Only One Litigation Client in 2025, He Was Previously Charged, Just Like the Serial Strangler From Microsoft (Whom They Now Represent)
Karma is superstition, regulators are not
Project 2030 to Cover How "Project 2025"-Styled Anti-Media Zealots From America Targeted Techrights and Tux Machines
The common denominator is also their attacks on women
Brett Wilson LLP Failed to Meet Deadlines Set by Judge 7 Months Earlier, Tried to Ruin Our Holiday, Then Had the Audacity to Ask Us for Over 3,000 Pounds for Its Own Lateness
As a matter of principle we will never respond to assassin while we are on holiday
On Claims That After Bluewashing Red Hat Will Increasingly Become an Indian Company
Discussed this week (long and detailed)
Americans Attacking British Sites Only Months After They Leave America
We find it kind of funny if not ironic that this site, originally an American site, got legal harassment only from Americans and only months after it had moved to the UK
Despite Losing Over a Quarter Million Dollars a Year Software in the Public Interest (SPI) Gives Helping Hand to Libreboot
SPI's financial state depends a lot on its public image or its reputation
Slopwatch: Google Helps Plagiarism and Sends Traffic to Ripoff Artists
That Google as a company helps spamfarms is noteworthy
If You Want to Know the Future, Listen to the Free Software Foundation (FSF) and Andy Farnell
We're sure the FSF will have plenty of its own output
Links 18/09/2025: A Taliban Ban on Internet Access and Troubled US Job Market
Links for the day
Gemini Links 18/09/2025: Computer Literacy and Accessing Alhena's Database
Links for the day
Links 18/09/2025: US War on Media (Truth Banned, Cancel Culture by the Hard Right), NYT Chief Executive Warns Cheeto is Deploying ‘Anti-press Playbook'
Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Wednesday, September 17, 2025
IRC logs for Wednesday, September 17, 2025