Bonum Certa Men Certa

Good News, Bad News (and Back to Normal)

Video download link | md5sum 948cc5f102085cd52f4d356b486c2586 Major Incident and Recovery Creative Commons Attribution-No Derivative Works 4.0



Summary: When many services are reliant on the integrity of a single, very tiny MicroSD card you're only moments away from 2 days of intensive labour (recovery, investigation, migration, and further coding); we've learned our lessons and took advantage of this incident to upgrade the operating system, double the storage space, even improve the code slightly (for compatibility with newer systems)

THE good news is that Gemini is expanding faster than we predicted earlier this month. Lupa is now just 11 capsules short of 2,000 and yesterday we received some E-mails asking about Gemini downtime (we also got some inquiries over IRC, which means the Gemini capsule really matters to people).



"These things are inherently fragile; telling people to reduce the number of write operations is almost unreasonable because what good is a system you cannot use (or program) as you wish?"So why was it down? The short story is, it was a hardware failure. Not the fault of GNU/Linux or anything like that (in fact, credit to GNU/Linux for letting us fetch another complete backup of the entire system despite the whole file system being in read-only mode). There was no panic, just frustration, and based on what we heard about MicroSD-based (for boot) systems such an error was inevitable and almost predictable. The latest backup (before the "emergency" one was initiated) had been marked only a few days old (contents at most a couple of days behind).

All the services are now back online, the operating system was replaced by Debian 11, and the machine has twice as much storage space as before, which ought to permit us to do things we didn't even dare when space was tight. To reduce future downtime I also bought a spare disk (card actually) and will work on improving/reducing D-R time, as it's likely that a similar incident will happen later this year or next year. These things are inherently fragile; telling people to reduce the number of write operations is almost unreasonable because what good is a system you cannot use (or program) as you wish?

"We're hoping that tonight and tomorrow we can make up for the lost time..."Debian 11 is quite nice, but of course imperfect (perception is an impossibility). It's the first time I use Debian 11 (my wife, my sister and myself all use Debian 10 on our laptops) and maybe I'll get to write some positive things about it some time later this year (once I gain more experience/s with it).

We're hoping that tonight and tomorrow we can make up for the lost time; I hardly slept yesterday (stayed awake for about 20 hours straight, then just 4 hours of sleep) and we have a bunch of things lined up that I never managed to publish as restoring services (like IPFS and Gemini) was more pressing a task, more urgent a need.

The hardest part (to me personally) was having to go to Town for replacement components, knowing that few shops still exist (even fewer because of the pandemic) and the bigger shops are full of unmasked people who don't respect people's perimeter (it's not helping that our government likes to pretend COVID-19 is just some past event).

Recent Techrights' Posts

Turns Out LLMs for Code Don't Save Time and Don't Improve Quality
Neither legal nor useful
 
Microsoft Repeats the Mistakes Made by the EPO After We Exposed a Major Microsoft/EPO Scandal 10 Years Ago
That scandal was all over the media, not just in English
The Demise of LLMs
We've just checked BetaNews again. They've dropped all the slop and went back to human authors.
Gemini Links 13/07/2025: Sonpo Museum of Art and FCEUX
Links for the day
Links 13/07/2025: UnitedHealth's Censorship Campaign, Australia Wary of China
Links for the day
Firing Away With Nonsense
Or fighting fire with fire
Links 13/07/2025: Climate Crisis, GAFAM Poisoning the Water
Links for the day
The Microsofters Will Have an Obligation to Compensate Us
This story isn't just about Microsoft. It's also about corruption, there are many women victims, there is abject "abuse of process", and many more scandals to be illuminated in years to come.
Reproducing at the EPO Instead of Producing Monopolies for Foreign Monopolies With Their Price-Fixing Cartels
Does the EPO recognise the need of well-educated Europeans to bear kids?
Valnet Inc. Dominates Real (Not LLM Slop) GNU/Linux Coverage in 2025
And likely in prior years, too
Free Software Foundation (FSF) Fund Raiser Goes on
Later this month we'll expose another OSI scandal
EPO Staff Representatives Issue a Warning About Staff's Health and Inadequate Care
Even the EPO's own stakeholders (money sources) are openly protesting against what the EPO became
Links 13/07/2025: Partly Assorted News From Deutsche Welle and CBC
Links for the day
Gemini Links 13/07/2025: Board Games and Battle Styles
Gemini Links for the day
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Saturday, July 12, 2025
IRC logs for Saturday, July 12, 2025
Plunder at the Second-Largest Institution in Europe
cuts, neglect, health problems, even early deaths
Links 12/07/2025: Political Developments, Attack on Opposition, Climate Actions
Links for the day
Gemini Links 12/07/2025: Melodic Musings and Small Web July
Links for the day
Links 12/07/2025: Jail in China for Homoerotica, South Korea Discriminates Against Old Workers
Links for the day
If Only Everything Was Rewritten in Rust, We'd Have No More Security Issues?
Nope.
Links 12/07/2025: Birdwatching and Fake/Misleading Wall Street 'Valuation' Figures
Links for the day
Gemini Links 12/07/2025: How to Avoid Writing, Apps for Android
Links for the day
Using SLAPPs to Cover Up Sexual Abuse and Strangulation
The exact same legal team of the Serial Strangler from Microsoft and Garrett already has a history fighting against "metoo"
EPO Staff Committee on Harassment in the Workplace
slides
Adding the Voice of Writers to UK SLAPP Reform
The journey to repair antiquated (monarchy era) laws will likely be long
EPO Takes More Money From Staff for Speculation (Pensions), Actuarial Study Explains the Impact
"The key change in this year’s Actuarial Study, due to cascading the new “risk appetite” from the financial study, is a significant increase of the total pension contribution rate of 5.7 percentage points, up to a total of 37.8%. This is driven by an unprecedented decrease in the discount rate of 105 bps down to 2.2%."
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Friday, July 11, 2025
IRC logs for Friday, July 11, 2025
Microsoft - Like IBM - Does the "Relocation" Tricks (Start Over Elsewhere, Then Get Sacked by Microsoft)
It is a "low blow" or a "dick move"
After the Free Software Foundation's Campaign to Raise Money Let's See Campaigns to Finish Off Microsoft (Vista 11, GitHub etc.)
Microsoft is in effect collapsing
Your Publications Have No Major Impact Unless or Until You "Get Some Heat"
we're on the right track
Slopwatch: A Cause for Hope, the Hype is Dying
For about a month we showed that becoming a slopfarm - for several weeks - resulted in utter failure and ruin for BetaNews
Links 11/07/2025: Censorship Worsening, 3D Printing Success Stories, UK and France Unite Around Nukes
Links for the day
Gemini Links 11/07/2025: Zorin OS and Scriptonite Updates
Links for the day
Links 11/07/2025: Hardware, Russia, and China
Links for the day
Links 11/07/2025: Intel Collapsing and Microsoft Resorts to Bribery to Push Slop Via Obligatory Education
Links for the day
The EFF Sided With the Team That Strangles Women and Tells Women to Kill Themselves
They say that apathy and inaction are a form of a "stance"
"Nat [Friedman] and [the Serial Strangler From Microsoft] Were Always Exceptionally Close," Says Former Housemate and Colleague
Now Alex (hiding behind another name when that suits him) not only attacks women but also people who merely report what he did to women
Exemplary List of Things That Are Not Artificial Intelligence or Even Intelligence
The "age of AI" or "era of AI" or "AI revolution" mostly boils down to rebranding, just like "the cloud"
New Letter From the European Patent Office Explains How the Office Plots to Grant Many Illegal Patents, a Self-Fulfilling Prophecy of 'Growth'
Open letter to Mr Rowan (VP1) and Mr Aledo Lopez (COO)
Abuse of Process
5RB is employing people who help violent men
What Microsoft's Nat Friedman and Microsoft Lunduke Have in Common
"Get in da car; No time to explain, loser"
Microsoft and IBM Don't Have Much of a Future (They Mostly Pretend at This Point)
IBM and Microsoft are in some ways alike but in many ways different
It's Not Just Twitter (or X.com) That's Dying, Microsoft's Equivalent is Dying Also
Unable to find a business model
GitHub Copilot Can Cause the Bankruptcy of GitHub to Come Sooner and GitHub to be Shut Down Just Like Skype
Some publicly available information suggests that even for each paid subscriber for plagiarism (LLM 'coding') GitHub Copilot still loses more money than it makes
Wayland is Bad for the Planet
If you use Wayland, it'll take you longer to accomplish tasks and you will consume more energy (or battery life)
Legitimising Those Who Sabotage You
Microsoft is a very malicious company
Over at Tux Machines...
GNU/Linux news for the past day
IRC Proceedings: Thursday, July 10, 2025
IRC logs for Thursday, July 10, 2025
On Microsoft Layoffs
we might be looking at about 60,000 Microsoft layoffs since 2023
EPO Management Already Breaks Its Own Promise (Lie) on "Bringing Teams Together"
This gut-punching move happened just 2 days ago
Gemini Links 11/07/2025: Occupation of 2025 and "Old Man Yells At Soundcloud"
Links for the day
Our Lawsuits Against the 'Cancel Mob' (Ringleaders) Helped Reduce Anti-Free Software Online Abuse
That's not to say that lawsuits are the best way to handle terrible people. But that can help.