A Code of Conduct Can Lead to Deterioration of Quality Control in Linux (Nobody Reprimanded for Technical Issues, Instead Critics at Times of Crisis Get Reprimanded)
THE pertinent details about the latest Linux issue are not fully disclosed just yet, but it has been assigned a rare level of severity. We need to discuss this as Linux grew exponentially larger and now contains multiple programming languages, so many veteran Linux coders cannot even comprehend what certain parts of Linux do. Especially parts that have not matured. Their freedom of speech (expression, opinion, criticism) is curtailed further, as the culture of Rust is imported to the core of the system (criticising Microsoft is like "hate speech").
Let's be clear upfront.
This is an ongoing and still-developing story about a critical issue impacting Linux and thus many GNU/Linux distros. Debian, which many distros are derived from, is also affected. We've thus rebooted the server (IRC downtimes are unpleasant; IRC has no redundancy, as it's not spread across nodes) twice in the past day, loading the latest kernel and checking a whole bunch of stuff.
In this short post we will write primarily about what's known (this is still discussed a lot in our chats, with the "knowns" separated from the "unknowns"). The media will soon follow up (not just some blogs), so we too can follow up in the future with further details. As a C programmer myself, I can make sense of some of the code and defects. The media won't even cover those aspects as nowadays "tech" gets covered by people who just parrot buzzwords like "AI", not proficient programmers.
Having studied the Debian repositories, mailing lists (and LWN comments about those) et cetera, it seems safe to say that the issue isn't just a real issue (unlike mere hype, one might say dramatisation) and some people are already impacted. One associate had started to see accounts of trashed systems before rushing to update ours.
How did it all happen, whose fault, and what could be done to prevent it? We'll probably have more definitive answers in days to come.
There are almost no details about the problem available, as an associate explains. Debian apparently pushed out the tainted kernel many days after the problem was known and since Linus Torvalds is not in charge of the kernel anymore (he has just said he's not a manager), we cannot see him saying a word about it. There's no word from Linus about this as of today; I checked LKML and messages from Linus going as far back as weeks ago.
When did he find out about this and what did he know? "I AM SORRY" won't be enough in this case; many critical operations around the world may have silently lost some data; some won't be able to assure system integrity or even boot the system. It impacts both servers and desktops/laptops. Devices (like portable phones)? Maybe... if they use ext4.
Just about a month ago Linus complained about bad code making it into the kernel at the last minute: "If people then stop testing new kernels because they think new kernels might break their setup, we have lost something truly important."
Now we have a severe data issue. It cannot be undone, even if a fixed kernel gets installed. People are encouraged to check their data's integrity against backups, if any are available (RAID is no panacea here).
"Perhaps he knew and could not speak for fear of stepping on some corporation's CoC," an associate hypothesises. "Even the package descriptions and info were unclear as to which actual version was being deployed."
Some years ago we saw bad kernels (seldom a stable kernel, usually an RC) breaking some hardware, e.g. putting monitors or network cards at risk, but those can be replaced. Data cannot be. It's a truly complicated issue.
Confidence in Linux is the main casualty.
"The sick part is that it appears 1) Debian backported the broken patch 2) Debian deployed said broken patch even after it was known to be broken 3) they left the package up in the repository for days before pulling it," an associate alleges.
There too many uncertainties still, but getting the Git history helps check what happened and when exactly. "The fact that it was in Bullseye points to (1), (2) would be a matter of looking in the Git repository, and [regarding] (3) we installed it from the repository."
The tracking of the package can be done via this so-called "webapp" or the tracker (also security tracker), which has changes in the Git file.
Why was the public not informed until a weekend? This isn't like a security issue that can be exploited by hostile parties if revealed "too early"... or it is? Could this issue be triggered remotely with some network-bound request, taking advantage of some particular weaknesses in ext4?
Tracing back the communications about this, (timeline in the message bodies at [1, 2, 3]), the key powwow can be traced back to 6 days ago, the 5th of December, but discussion is going back to late November. We know that linux-image-6.1.0-14-amd64 was removed during the weekend from the package repository of Debian, but where was an official statement on the matter? There's still no official communication about it.
Torvalds used to be outspoken when bad code and bad practices were adopted and spread throughout the team. Now he says absolutely nothing and there's no sign that negligent/reckless acts will bear any consequences; instead, Jim Zemlin will blame the "opinionated" community - a term that he does not even understand and represent something he is hostile towards.
Quality control demands opinionated people, even blunt opinions at times.
In his latest "talk" (chat with Dirk) Linus admitted he no longer writes real code (just pseudocode portions) and that he just does a smell test of code he allows into his tree. He also admits he cannot understand Rust (but is learning a little as he goes along), so he essentially permits random people to toss large lumps of code without a vendor-neutral (independent from the vendor of origin) audit - i.e. an additional problem.
What will be done to ensure the above does not happen again? Rewrite the file systems in Rust? That would make things worse and less predictable. █