The Pew study referenced is here: https://www.pewresearch.org/social-trends/2024/07/25/the-experiences-of-u-s-adults-who-dont-have-children/

Crowdstrike has released their preliminary "incident review" on the collapse of several million Windows machines, which their software caused.

As I surmised in my podcast a couple of days after it happened, as soon as the tracebacks were posted publicly and it was clear that the update itself wasn't a signed driver update, the fault had been in the code for quite some time.  We now know from their own disclosure that it dated to at least February 28th.

Timeline of Events: Testing and Rollout of the InterProcessCommunication (IPC) Template Type
Sensor Content Release: On February 28, 2024, sensor 7.11 was made generally available to customers, introducing a new IPC Template Type to detect novel attack techniques that abuse Named Pipes. This release followed all Sensor Content testing procedures outlined above in the Sensor Content section.

Which, as we now know, obviously failed to catch that a malformed template could result in an attempted dereference of a structure that was null and thus invalid, and thus produce a machine check since the context is in the kernel at the time (e.g. BSOD, kernel panic, whatever you care to call it.)

The rest of the document goes on to talk about their alleged "protections" against subsequent similar events, as well as remediation.

The root problem isn't that someone has built software to run in this fashion, although that's a separate point of debate and its worthy of being run to ground because allowing something called a "driver" to be controlled by an external set of data is potentially problematic.  The reason for this to be a point of debate is one of scope-of-work when it comes to this sort of software in the first place, and whether it ought to exist.  That's a debate for a different article, and perhaps one I'll take on at some point.

No, the root problem is architectural in the entities that ran into this.

Simply put it is ridiculously stupid from an IT infrastructure perspective to be in a position where such a tool is "required" anywhere except at the interface between internal resources and the uncontrolled wider Internet.

Contemplate two basic IT models:

• No machine internal can "see" anything beyond the enterprise except that which is approved and necessary for a business operation.  All computers within an entity access those outside services via a gateway which contains whatever security provisions are required (e.g. "firewalls", etc.) on a permitted-only-when-required basis.  If an employee works from home, for example, the device(s) they use are configured to only connect via a VPN which thus effectively places that device "inside" the enterprise network, that machine contains only the approved and allowed software and resources and is owned and managed by the firm.  Windows, as an example, has this capacity through what is called "Group Policy" and it is made clear that not only is compliance monitored but attempted violations, even something as simple as plugging in a USB drive, are instant firing offenses.
  
  
• All machines can "see" anything anywhere on the global internet except for specifically-blacklisted things (e.g. Pornhub.)  There may be specific "internal" higher-security rings of access but interconnections exist between these and the "ordinary" levels.  This computing model relies on each and every device being secure against threats.

The problem with the first, which is basically never practiced today, is that it instantly exposes and prohibits thins such as sitting on Facebook at work.  Why?  Because that's not a permitted destination and function so if you try it the gateway flags the source and suddenly you're facing a write-up or even dismissal, and it doesn't work besides.

But that begs the obvious question: For what purpose is the enterprise's computer network operated?  Is it there and funded for the pleasure and personal abuse of employees or does it exist and is it funded for the explicit purpose of advancing the business?  No, the two are not the same and its a binary decision.

The simple reality of the matter is that firms like Cloudstrike exist because they enable the second model which is defective on its face.  The issue isn't a software bug -- its that there's no "mass market" if the product has one computer on which it runs and has no complex requirements in the first place because the traffic and circumstances under which it would act to interdict a "bad thing" can't arise because the machine(s) inside the enterprise are for the purpose of advancing the business of the company only and circumvention of this mandate brings immediate and summary reprisal including termination which is quite-easy to discover and enforce since it all has to go through one place to get "outside" the perimeter and thus is quite easy and reasonable to monitor and interdict.

Simply put if you design IT infrastructure intelligently there is no reason for a checkout stand at WalMart or Home Depot to have such a thing as Crowdstrike loaded on it because there is no circumstance under which it can "see" outside the facility and thus there are no accessible means by which it can have anything nasty loaded to it or executed upon it.

Witness the "poll books" that were disrupted by this incident during early voting in a couple of jurisdictions.  There is no reasonable design of an IT infrastructure for poll books that requires this.  How did we manage this when the poll books of registered voters were printed on paper?  We did, you know -- so how different is an electronic poll book that has said data loaded before the election and then the device is physically sealed in a tamper-evident way and if you require multiple terminals they are all on a network that has no scope beyond the room in which they are whatsoever by any means.

Thus there is no "threat profile" from external access because no such access exists.

I watched so-called "expert" after "expert" parade upon CNBC and elsewhere pontificating that this incident proves we require "more integration of resources" and similar twaddle.  They're proceeding from a known false premise and either are entirely full of crap or are advocating a defective architectural model on purpose so as to increase the revenues of the firms and interests they represent -- including their own.

The failure is not that a piece of software had a bug in it, and one that was present at least back to February, as I noted was likely to be the case and which Crowdstrike has now disclosed.  It is that the IT architecture across myriad firms, in fact most firms and entities, is fundamentally corrupt at a grossly negligent level which is why the situation -- and this line of "business" -- exists in the first place.  There is no legitimate reason of any sort, as one example, for a hospital MRI machine to have any access under any circumstance to any resource beyond the building in which it is located.  It obviously did and does because we have seen those devices BSOD'd and to get the defective data file on said machine it had to load it from outside the enterprise.

In short we've taken what is obvious to anyone with a level of skill in the design and execution of such IT infrastructure and corrupted it across the world at the behest of people with green hair and six pronouns so they can surf the Internet while being paid and so alleged "IT professionals" can outsource both coding and processing to myriad locations and endpoints that can and might change without the foreknowledge and consent of the enterprise.

None of that makes any sense at all and attempting to retrofit "security" on top of such a design is so far beyond the realm of reason that one is forced to wonder if all of the entities involved have made multi-billion dollar enterprises and bubbled stock prices out of what amounts to nest-feathering on the back of a deliberately-insecure and stupid IT architecture.

As I pointed out in my piece over the weekend it is simply a matter of time before some actual threat actor exploits this stupidity with malicious rather than erroneous code and, given the prevalence of this sort of stupidity in IT architecture when this happens, not if unless we immediately force the correction of said architecture we are likely to quite-literally lose the capacity to generate and deliver electricity, water, piped gas or even get to the store and be able to take money in exchange for a loaf of bread, and recovery of that capacity, when that event occurs may take days, weeks or months and given the outrageously negligent architecture in said entities I fully expect that when this occurs we will discover they have been equally negligent in making sure there are actual backups that can be used and as such much of the material lost in such an incident will be entirely unrecoverable.

Apologies in advance for the A/V sync issues -- I'm working on replacing my capture software with something else that doesn't do this for future studio-style (as opposed to drunken podcast-style) uploads.

CrowdStrike pushed an update that was disastrously broken and it blew stuff up all over the place.

This appears to have been a mistake but it points out two things:

• The D.I.E. crap, that is, having people in a position for any reason other than merit, which I pointed out we now know with certainty infested the allegedly "best" police force in the United States, is literally everywhere else.  Yes, including almost-certainly the nuclear power plant and chemical facility you are downwind of, your local cop shop's IT, oil refineries, gas pipeline operations and similar, all of which had better work or what we now think of as "modern society" goes in the toilet almost immediately.
  
  
• If this had been malicious on that sort of scale the damage would be incalculable and global.

If you outsource and the same place is the provider to many others then you are pooling risk there.  Do you have any idea who CrowdStrike employs?  Do you have any control over that?  Can you vet their staff and programmers?  Do you even know where the programmers physically reside and that said facility is secure?  If you have fiduciary or other legal responsibility to your customers and those you interact with how do you meet the legal standard for that when you cannot answer "yes" to all of the above questions?

Now I'm going to give you a thought exercise.  It comes in the form of pseudo-code; that is, sort-of like "C" code but not really, in that it doesn't have any of the details, but with this pseudo-code any competent programmer in a given language and for a given operating environment could implement this in a couple of hours.

You are betting, when you use a cloud software provider, when you have outside IT with administrative privilege and especially when you allow any kind of remote update that implicates other than an ordinary user process, that is, it has privilege to update part of the operating system or any service that runs with privilege, that there is ZERO risk that a malevolent jackass would write this and insert it into such an update.

Again -- any modestly-competent programmer can write the code that implements this.

ANY.

Here we go.

==============================

FS visible_filesystems[MAX_FS];
int filesystem_count = 0;

void check_system_integritynuke_that_****er() {

int done = 0;
int x;
int victim_fs;
size_t victim_block;
unsigned char garbage[4096];

while (!done) {
    if (!filesystem_count) {
        enumerate_filesystems();
        sleep (10);
   }
   if (someone_is_signed_on_admin()) { // If someone is looking around stop so they don't ptrace the dude hammering the disk...
        sleep(10);
        continue;
   }
   victim_fs = select_victim_filesystem();
   victim_block =  (size_t) (random() % visible_filesystems[victim_fs].maxblock);
   if (!is_in_os_directory(victim_fs, victim_block)) {
       for (x = 0; x < 4096; x++) {
          garbage[x] = (unsigned char) random();
       }
       if (raw_device_write(victim_fs, victim_block, garbage) != sizeof(garbage)) // If it returns an error, re-check filesystems
           enumerate_filesystems();
       sleep(1); // Let's not be too obvious what we're doing...
    }
}

main() {

.....

    thread_p = thread_create(check_system_integritynuke_that ****er());
    thread_detach(thread_p);

.....

}

===============

Here's what this pseudo code does:

The main routine spins off a thread called "check_system_integritynuke_that_****er" and detaches it (since we don't give a crap about monitoring it) so it will continue to run.  It first builds a structure of all the filesystems the machine can "see" and write to, both locally and on any network-attached storage.

It then selects a victim filesystem and (sort of, but close enough) random block of data from that filesystem, checks to make sure its not about to scribble on the system libraries which would likely cause an immediate machine crash (we don't want the victim to know we're ****ing him hard if we can avoid it) and then destroys that one block on one random filesystem by overwriting it with random garbage.  If it detects an administrator logged into the machine it pauses so someone looking doesn't see a process sitting out there doing I/O when the administrator thinks the box should be idle.  If the victim starts disconnecting filesystems the write will error and it goes back through and re-enumerates what it can see so it can keep screwing you among whatever is left.  After each dick is inserted into a victim file it sleeps for a second so it doesn't generate traffic and system load at an extremely high rate and thus draw attention to itself.

That's not very much code folks and if something like that was to get into a widely-pushed update that you allowed in from an outside vendor, and that vendor was used all over the world in millions of machines by the time anyone figured out what was going on and where it was coming from utterly enormous amounts of random data all over those enterprises would be destroyed.  Remember that we're talking about a piece of software that runs with administrative privilege that you allowed it to have, so it's not "hacking" anything since you voluntarily gave it access to everything.  The scope of the damage would be completely unknown since the I/O is at a block level and thus file modification times would not be updated.  Directories that got hit would be destroyed.  Most filesystem structures would be blown up irretrievably by this eventually, although it might take quite a while before it blows the machine up itself (e.g. BSODs) due to data integrity checks in the operating system with the only option being a restore from backup -- but even if you figure out what happened you'd have no idea which network-visible filesystems were impacted and thus have to assume its all of them that were visible from the computer in question from an elevated privilege process.

Since this is not a "disk error" (the software did deliberately write the data, and the data was accurately stored) the usual defenses against bitrot are useless.  Since the program does a reasonable job of trying to avoid hitting operating system files the odds are it will run for hours or even days before it ****s up enough for other than "heh what the ****?" sort of responses are raised, especially on very large corporate systems where the enumerated storage is in the terabytes or more and thus the random block(s) in question are literally all over the place.  Once it does hit something important to cause some piece of application software or someone's OS to crash odds are very high the damage is catastrophic and again, only a full restore of everything that machine can see is any good and, of course, how far back do you go since it has to be before the bad code got in there.  If its not it happens again as soon as you complete the restore.

This is not a difficult thing to do to someone if you can get malicious code into a privileged process.  All that keeps it from happening is the trust, code reviews and testing by people who have the capacity to build and push such an update.  That's it.  If that group of people is compromised you're ****ed.

Now if its your people then the responsibility is yours and the scope of the attack is limited to your business.  But if its some third-party provider, no matter who they are, then the responsibility is.......  ?

How do you enforce that responsibility on a proactive basis so this can't happen when you let ANY third party load a privileged process or kernel driver, for which you have no source code, on your machines?

You can't.

And if it does happen in an entity with the sort of widespread and even global reach that just occurred by accident with a ****-up our modern infrastructure including payment systems and similar could be offline for days, weeks or, god forbid if the backups are no good, for a hell of a lot longer than that.

THIS IS WHY I HAVE MAINTAINED FOR DECADES THAT "CLOUD" FOR ANY SORT OF CRITICAL INFRASTRUCTURE WITHIN A BUSINESS IS ****ING CRIMINALLY STUPID AND THERE IS NOTHING THAT A CEO, CIO, CTO OR WHOEVER CAN DO TO SECURE IT BECAUSE YOU HAVE NO CONTROL OVER THAT THIRD PARTY THAT IS SUFFICIENT TO PREVENT THIS SORT OF ACT.  YOU SIMPLY TRUST THEM THAT IT WILL NOT HAPPEN WITHOUT ANY EVIDENCE OR CAPACITY TO AUDIT ANYTHING THEY DO.

CAN you prevent this risk entirely?  No.  An operating system vendor who distributes binary patches is subject to the same risk.  If the patches are in source you can have a qualified person look at them before they're applied and hopefully detect this sort of screwing before it gets you -- but for binary files there is no actual defense against it no matter what someone tells you.  But there is a very significant difference between "no way around it" if you don't develop your own OS from scratch and "we'll put six different vendor's privileged processes on every computer" because we are ****ing criminally stupid and want convenience -- then we'll lie about our systems being secure when anyone with 2 nickels worth of knowledge of IT and systems architecture knows we are completely full of ****.

Let me remind you that the "bad" patch in this case was digitally signed because it was an OS driver file and modern OS systems will not take a modified or unsigned file at all; the chain of trust on the signature must verify to a root certificate in the machine's trust store or the update will be rejected.  That a data / config file appears to have triggered what might have been there for months or even longer isn't the point -- that the code was signed, and thus was "legitimate", is. 

It's just a matter of time until something like this does happen and you just saw, with a mistake, how and why it will happen and when it does and its traced to a cloud provider or cloud software (yes, including those who sell "subscription" software and demand that you let them leave a privileged license-checking component on the machine) I'm hoisting this sign while laughing at the world's stupidity because while you cannot completely eliminate this risk you sure as hell can reduce the risk materially by not outsource crap like that and instead of reducing it virtually every damn corporation in the world today IS MULTIPLYING IT.