Computer says no

[Simon]

This comment probably has an audience of only one or two here, but I'll say it anyway.

Our cloud edge platform has about 450,000 machines or so. When an update goes out it doesn't go out in one big hit, it first goes out to a small core set of machines (zone 1). Automated testing then takes place to see if the update is sound, those machines are still working correctly, and errors are within threshold. It then goes out to zone 2 (a larger group) and the whole testing process repeats. Finally it goes to zone 3 (the rest), and again the same automated tests are done.

There's just no way that such a disastrous update could go out for us in the same manner. It's mind boggling that 8.5 million machines or so were impacted and there was no error checking along the way to pause the distribution.

Also staggering that their CEO forgot to apologise on TV.

[DeskJockey]

I guess it boils down to the difference between companies that are an online service and those that use them because they have to.

[jamcg]

Also staggering that their CEO forgot to apologise on TV.

Possibly at request of the board and shareholders because apology could be interpreted as an admission of liability which they won’t want to do as the amount of claims for compensation coming their way shortly will be enormous

[Beany]

A brief technical breakdown with a bit of background on how Windows handles kernel mode drivers, from a former Windows dev

[GG.]

Had a fun day yesterday trying to get back from Naples. The flight that should have left at 12.15pm (and was actually on the ground waiting) couldn’t take off as most of Naples ground coordination, the tugs, baggage handlers etc is run on Windows based machines. We eventually got pushed back at 4.48, at 5 pm Jet2 had made the call to ground the flight and put us up in hotels as the crew were passed their hours so it was then a race to get in position on the runway, not easy as we we parked at the far side of the airport. By my watch we lined up at exactly 5pm but at that point the Captain, (a former Red 1 in the Red Arrows no less) was clearly going regardless and did promise us that he would take the aircraft as fast as it would go and managed to make up 30 minutes. Of course we knew all the trains were all cancelled and a bus replacement service was running, 6 hours, no thanks so a £200 taxi it was. Thankfully he arrived with Kronenbourg and Stella. My nephew is still in Naples as he flew with Ryan Air and well, Ryan Air.

We were also travelling on Friday - back to Stansted from Krakow. We had a near 4 hour delay on the flight and lots of stress with huge check in queues at the airport (needless as it turns out give the delay) but overall were very lucky.

I think having a flight at 5.45pm helped as the airlines had kind of got their shit together by that point. We were also flying with Ryanair which I never do (but flight times/days dictated this time) and at least in Krakow airport they did a very good job and had all hands on deck with 6 check in desks open to process the 4 or 5 flights they had lined up. Travelling though Stansted on a weekday early morning during summer was like a special type of freak show however - I've never seen as many neck/face/head tattoos or orange people with lip filler in all my life. Made Gatwick airport look like a walk around South Kensignton

[dinny_g]

Travelling though Stansted on a weekday early morning during summer was like a special type of freak show however - I've never seen as many neck/face/head tattoos or orange people with lip filler in all my life. Made Gatwick airport look like a walk around South Kensignton

[Matty]

Also staggering that their CEO forgot to apologise on TV.

Have you seen what happened in Mr Robot Season 1 when a CEO apologised live on air for an IT outbreak? Probably for the best.

[Beany]

Update - Crowdstrike have released an interim report on what happened - summary here:
https://www.theregister.com/2024/07/24/ ... r_failure/

Basically, nope, wasn't a definition file but a separate update channel that's not user controllable, for sensor config - and their release management/QA system failed to catch the bad update as it went out.

Apparently they're gonna add user controllability to that (IE like they have for definition deployment), and - heaven forfend - release notes.

So er, that's not gonna fill many people with confidence - apparently their release management process either doesn't include much in the way of "spin up a Windows box, install Crowdstrike Falcon, apply latest update, see if it reboots OK", or it failed entirely, and you'd think testing of that process to make sure it's not gonna cause millions of dollars of damage if it goes wrong would be kiiiiinda important in a piece of software that operates in KernelMode/Ring0/as a god.

[Mito Man]

Have you seen their apology? A $10 Uber Eats gift card

[Simon]

Have you seen their apology? A $10 Uber Eats gift card

And lots of reports of the code not working....

[Beany]

Cyber security company sends out what looks suspiciously like a phishing email.

Fucking slow clap there, lads.