A Week After a Worldwide Windows Outage Microsoft is 'Bricking' Windows All On Its Own, Cannot Blame Others Anymore
A look back at a week of lousy press coverage, Microsoft deceit, and lessons to be learned
A week ago the "world stopped" (the world increasingly depends on digitalised systems to operate properly) because Windows systems had broken down. Since then, Microsoft's evolving "blame game" was the subject of scorn and criticism and then, about a day ago, Microsoft had nobody to blame but itself.
It is very good to have waited a whole week. Intentionally, too. We wanted to see the dust settling somewhat, then take a fresher look (back). Today is the day to debunk some common myths and show examples.
As an associate put it, "there are some kinds of news which take time to investigate and get to the bottom of. One thing though was Microsoft was protesting so much that one could almost guarantee they were trying to cover for several problems."
There were problems other than a bad patch. The media lost sight of these.
First, a little background though.
A few days ago in the FSF's mailing lists we saw this message by Akira Urushibata on "Microsoft and Crowdstrike". It's a decent summary:
The massive computer outages of Friday July 20 were caused by an update of security software. Microsoft computers were affected but the source of the problem was an update supplied by a third-party security firm named CrowdStrike. Headlines showed the names of both Microsoft and CrowdStrike. I believe this was necessary because only computers running Microsoft software were affected: the information was valuable for those dealing with the problem. However the wording probably led the unsuspecting to believe that both firms were responsible. The articles which followed the headlines typically said that Microsoft had identified the problem and was helping those affected.I would like to know about the relationship between Microsoft and CrowdStrike. It is quite likely that security software requires privileges that the OS maker has to provide through a special agreement. The two companies are separate, yet they are likely in a cooperative relationship of some sort.
If Microsoft was not at all responsible for the ordeal, how do we explain its generous attitude? One possible explanation is that it understands that it does not dominate the OS field any more. Customers are likely to consider alternatives including GNU/Linux after this ordeal and it would not be good policy to be callous toward their anxiety.
I would like to hear your opinions. Thank you in advance.
---
CrowdStrike and Microsoft: What we know about global IT outage - BBC News
https://www.bbc.co.uk/news/articles/cp4wnrxqlewo
This is an example of an article from a major news article that says that Microsoft is working on mitigating the issue, without reporting whether it has said anything about its own responsibility.
Two days ago Denis 'GNUtoo' Carikli responded as follows:
On Sun, 21 Jul 2024 10:13:55 +0900 (JST) Akira Urushibata <afu@wta.att.ne.jp> wrote: > I would like to know about the relationship between Microsoft and > CrowdStrike. It is quite likely that security software requires > privileges that the OS maker has to provide through a special > agreement. I understand this outage a bit differently.
Personally I don't think that the company names are very relevant here. As for the kind of software (anti-malware): it needs to have the most privileges possible to work properly as that kind of software is supposed to detect things like rootkits.
To make such an outage you need: - A lot of people and/or organizations to rely on some software that is updated automatically. - To have the producer of that software issue an update that prevent computers from booting (you don't necessarily need privileged software for that, bad luck and a bug, let's say in a filesystem driver for instance, could trigger that too, but if it's more likely if the software is privileged already). The update could either be malicious or be an accident.
It could even happen with free software in the future if some (new?) free software businesses follow a business model that has all these ingredients.
Now, with free software and the distribution model (what you get with a regular distribution like Trisquel, and that you don't get with Appimage / Flatpak), the update of a software (like xz for instance) doesn't happen instantaneously, and the maintainer(s) of a given software (like xz) cannot force users nor distributions to install the latest update.
So that leaves a lot of space for testing and for finding issues, so in case of issues not everything goes down at the same time, and some people/organizations will often find the issue before others.
The bigger issue: ----------------- Note that more broadly the free software distribution model differs a lot from nonfree OS, Android, or things like Appimage / Flatpak, so even extremely basic threat modeling can differ a lot.
A key difference is that in the distribution model, applications are basically trusted not to be malicious, and a lot of security systems / features are built around that assumption (the privilege drop, or hardening at compilation time are good examples of that).
This reduces a lot the attack surface. If we look at Microsoft Windows instead, there people download and run random binaries, so the attack surface is way bigger and too complex to really secure in practice.
And if you look at the Iphone instead, in practice it practice manages to remove all users freedom (you can't even run the program you want there without Apple allowing it) without even managing to guarantee users's privacy due to the business model of many applications in the appstore.
The issue is that the more we follow a model where we basically give all control to the developers of applications, the more we are exposed to issues that plague these operating systems (Windows, IOS, Android).
And the only thing in the way with this model is probably the sandbox, and the fact that not everybody runs the same piece of nonfree or badly written software.
For instance you could in theory have an application that for some reason become malicious (it already happened to some libraries packaged with NodeJS), and is updated automatically (this is by design), and manage to escape sandboxing (it only needs 1 exploit, that is not trivial to do though), then exploit buggy out of tree (free or nonfree) WiFi drivers or nonfree firmwares (that is probably easy to do), exploit bugs inside nonfree UEFI (that is probably easy to do but probably doesn't scale well to a big variety of devices) and completely take control of the computers at a very targeted or large scale.
And then if that starts happening, you might be tempted to start relying on the same kind of security mechanism nonfree operating systems use to avoid such issues (secure boot, remote control from the company that manages the operating system / app store, using anti-malware software that detect threats, etc). Not only this approach doesn't work well for users freedom in practice, but then you also end up being vulnerable to incident like the Crowdstrike one you mentioned.
So the only solution I know to avoid all that mess is probably to use what works well: free software, ideally of good quality (to limit both the attack surface and the need to always update), and to get it though some distribution (like Trisquel, Guix, etc) that don't give all powers to the developers of applications / OS components.
As for updates some distributions (like Guix or Trisquel) also provide tools to track CVEs, and some use backported security fixes (like Trisquel).
And also a good practice if you run some infrastructure with it is also to reduce the attack surface as this tend to work well (not run services that you don't need, etc).
Denis.
Yesterday the FSF wrote about it also:
Let's be clear: in principle, there is nothing ethically wrong with automatic updates so long as the user has made an informed choice to receive them. For instance, it's perfectly understandable that a public library might not want to pore over kernel changelogs; they simply want to receive the update and move on with their work. At the same time, software bugs happen. Free software developers know this better than anyone. The Linux(-libre) kernel does not have some mystic immunity to them. What our community does have is a social structure that, most likely, would have rectified the situation swiftly.What free software offers is a diversity of choice. Although we can understand how the situation developed, one wonders how wise it is for so many critical services around the world to hedge their bets on a single distribution of a single operating system made by a single stupefyingly predatory monopoly in Redmond, Washington. Instead, we can imagine a more horizontal structure, where this airline and this public library are using different versions of GNU/Linux, each with their own security teams and on different versions of the Linux(-libre) kernel. For example, a library in Vietnam wouldn't necessarily be dependent on an American software company for their day-to-day work.
As of our writing, we've been unable to ascertain just how much access to the Windows kernel source code Microsoft granted to CrowdStrike engineers. (For another thing, the root cause of the problem appears to have been an error in a configuration file.) But this being the free software movement, we could guarantee that all security engineers and all stakeholders could have equal access to the source code, proving the old adage that "with enough eyes, all bugs are shallow." There is no good reason to withhold code from the public, especially code so integral to the daily functioning of so many public institutions and businesses.
Those are Free software perspectives on the whole thing.
psydruid (in IRC) said that the Microsoft response became, "this can totally happen to Linux too, so it doesn't make a difference whether you run Windows or Linux"
psydruid said this was "a great way to shut down the discussion and it's so transparent too".
We saw many dozens of bad articles, almost 100 in English alone. We cannot respond to all of them (it would not scale), but instead we'll take a subsample. There are many overlaps there anyway.
Last week we saw this article saying "Microsoft deployed hundreds of engineers [sic], experts [sic] to restore services," in effect misrepresenting Microsoft as an authority rather than the cause of the problems, as an associate noted.
And apropos lying and lies, the associate said, this Microsoft-funded site repeated a lie. As the associate explained: "It is a cybersecurity breach because it affected Availability of said systems. Silicon Angle and the other minions of Microsoft are lying about the scope of the incident in that way, which calls into question the rest of their coverage."
He said "it exemplifies the lies and talking points that most of the other sites a now also taking up and peddling in place of investigative reporting."
"It should not escape notice that even though the claim is 'only' 8.5 million systems, those 8.5 had been installed in mission-critical locations by bad actors usually on the inside of the victim institutions yep it's still too soon to bring all that up."
Separately he said that "the spin of the Windows + CrowdStrike collapse can be addressed sooner. It is very important that their roles in the causing problems retain the spotlight and that it not be spun as an "IT" or a "tech" thing. Those are the headlines this [past] weekend."
"I'm seeing dozens of such articles in this batch and none that are not spin any more. Microsoft is also conflating server with desktop -- again. That can be addressed now without naming Microsoft, since you have an ongoing series about market share. Desktop is a separate market from the server market (also the mobile market, and the super computer market) and it would help prepare things to establish just how minuscule Microsoft market share is in regards to servers. Not just the physical numbers but in regards to the number of services. Remember that on a normal server, the ratio of services to hardware is many to one. On Windows gimmicks, the ratio of services to hardware is one to many. Inverted."
"Does "1% of all Windows machines worldwide" equate to "100% of all Windows servers worldwide"?"
As this one blogger put it: "The company did release a statement with “Technical Details ”. This is a big nothingburger. They are confirming what we already knew. Nothing is said about the server side, root causes and the chain of process failures that led to this incident."
A later talking point developed at the start of the week and was brought up again hours ago in IRC (so it's not a dead talking point yet). It's the "rumor (Twitter) spreading the idea that 2009 enforcement of anti-trust laws by the EU are to blame."
Microsoft tried shifting attention to many parties, including the EU and "Linux". In the above, it has been noted, "the account behind post 38 is probably some kind of astroturfer or professional troll, based on established posting history [and further to] the disinformation in comment #38 [...] Microsoft is flailing about grasping at straws for any kind of distraction: "EU gave CrowdStrike keys to Windows kernel, Microsoft claims". [...] Tom's Hardware now in on the disinformation: "Microsoft's EU agreement means it will be hard to avoid CrowdStrike-like calamities in the future" (that was about 5 days ago, only 1-2 days after it had all started).
And "apropos CrowdStrike," the associate said, "aside from the general failure of Microsoft products and their lack of suitability for any given environment, there is also the problem of desktop monocultures, as exemplified by this crap."
The outages resulted in fatalities, but the media probably stopped short of saying that for fear of being sued (as it would get Microsoft sued by the victims' families, seeing how Windows failures at medical facilities resulted in deaths). Not just BSoDs but actual deaths happened.
As an associate puts it: "The way of thinking which locks products and services to closed protocols and closed formats allows the creation of deadly monopolies. (See earlier very old articles on Microsoft as a national security threat in that regard.)"
"Few to none of the articles point out that there are other systems than Microsoft and that these other systems remain unaffected. Apropos other systems the London Stock Exchange still runs Linux, IIRC? If so, that would be basically the only reason it is still up and running."
"Day 2 of the CrowdStrike fallout sees what looks like a concerted effort to spin the problem as a 'tech' or 'computer' problem rather than something caused by and for Microsofters."
"Microsoft was named in the initial round. The second round is covering for them. The third round will likely be a repeat of spinning all praise for other systems or, worse, open standards and formats as purely schadenfreude only. Same ol' Microsoft, same ol' Microsoft media playbook."
"Availability is a key component in the standard definition of security. So by normal definitions these two incidents are major security incidents."
That's partly in response to "CrowdStrike CEO George Kurtz said “this is not a security incident..."
"The CEO lies through his teeth," the associate said. 'Security = availability, integrity, and confidentiality. CrowdStrike + Windows destroyed the availability aspect, and through NTFS collapsing due to unclean shutdowns, the integrity is going to fail too (Schade raised that last point in his blog today)."
Giving more examples of deficient or poor press coverage, this one is - as per the associate - "spinning it as a 'tech' or 'computer' problem rather than a dual problem of desktop monopolies compounding a larger problem of unfit software."
There's also "Global tech [sic] outage eases after widespread disruption, new focus seen on risks" (it's Windows, not "tech").
The LA Times said "Faulty software update causes global havoc for airlines, hospitals, governments", but this "confuses common with popular and confuses desktop with all computers," the associate noted. The New York Times made similar errors as "security is not an aftermarket add-on," the associate said, and "this incident drives that fact home".
Here is evidence of the deaths caused. However, it is "more spin, while Windows kills," the associate said. This one was a "good title," however "empty article", the associate said, instead promoting this piece ("The XZ advantage over CrowdStrike").
"Headlines ought to be reading: "CrowdStrike outage: Firms rush to adopt Linux and drop Windows"," the associate said, "and quotes ought to be reading: "... noting that the issue behind the outage really was security incident severely affecting availability if not also data integrity."
Microsoft-connected sites were so desperate to change the focus to "Linux" as Azure had gone down (as usual), ransomware impacted Windows a great deal (people lost sight of that because of the outages). A side effect of all this is that people are losing sight of yet more Microsoft failures.
I myself expressed my opinion mostly in editorial comments in the sister site (as the press went along), so many of the above comments aren't mine. That's fine. A plurality of interpretations helps too.
What can be learned from all this? First, Microsoft refuses to accept accountability, no matter what. It then tries to darken the reputation of the alternatives that are also the solutions. As for the media, it is either corruptible or inflicted with cowardice, and moreover it's too lazy to properly scrutinise false claims or investigate the facts. Parroting perceived authority is so much cheaper.
So people carry on dying needlessly and Microsoft blames "EU" and attacks "Linux" (neither of these caused hospitals to cease operations). █
