This comment probably has an audience of only one or two here, but I'll say it anyway.
Our cloud edge platform has about 450,000 machines or so. When an update goes out it doesn't go out in one big hit, it first goes out to a small core set of machines (zone 1). Automated testing then takes place to see if the update is sound, those machines are still working correctly, and errors are within threshold. It then goes out to zone 2 (a larger group) and the whole testing process repeats. Finally it goes to zone 3 (the rest), and again the same automated tests are done.
There's just no way that such a disastrous update could go out for us in the same manner. It's mind boggling that 8.5 million machines or so were impacted and there was no error checking along the way to pause the distribution.
Also staggering that their CEO forgot to apologise on TV.
Re: Computer says no
Posted: Sun Jul 21, 2024 10:04 pm
by DeskJockey
I guess it boils down to the difference between companies that are an online service and those that use them because they have to.
Re: Computer says no
Posted: Mon Jul 22, 2024 7:05 am
by jamcg
Simon wrote: Sun Jul 21, 2024 9:51 pm
Also staggering that their CEO forgot to apologise on TV.
Possibly at request of the board and shareholders because apology could be interpreted as an admission of liability which they won’t want to do as the amount of claims for compensation coming their way shortly will be enormous
Re: Computer says no
Posted: Mon Jul 22, 2024 8:56 am
by Beany
A brief technical breakdown with a bit of background on how Windows handles kernel mode drivers, from a former Windows dev
Re: Computer says no
Posted: Mon Jul 22, 2024 9:16 am
by GG.
Zonda_ wrote: Sat Jul 20, 2024 10:22 am
Had a fun day yesterday trying to get back from Naples. The flight that should have left at 12.15pm (and was actually on the ground waiting) couldn’t take off as most of Naples ground coordination, the tugs, baggage handlers etc is run on Windows based machines. We eventually got pushed back at 4.48, at 5 pm Jet2 had made the call to ground the flight and put us up in hotels as the crew were passed their hours so it was then a race to get in position on the runway, not easy as we we parked at the far side of the airport. By my watch we lined up at exactly 5pm but at that point the Captain, (a former Red 1 in the Red Arrows no less) was clearly going regardless and did promise us that he would take the aircraft as fast as it would go and managed to make up 30 minutes. Of course we knew all the trains were all cancelled and a bus replacement service was running, 6 hours, no thanks so a £200 taxi it was. Thankfully he arrived with Kronenbourg and Stella. My nephew is still in Naples as he flew with Ryan Air and well, Ryan Air.
We were also travelling on Friday - back to Stansted from Krakow. We had a near 4 hour delay on the flight and lots of stress with huge check in queues at the airport (needless as it turns out give the delay) but overall were very lucky.
I think having a flight at 5.45pm helped as the airlines had kind of got their shit together by that point. We were also flying with Ryanair which I never do (but flight times/days dictated this time) and at least in Krakow airport they did a very good job and had all hands on deck with 6 check in desks open to process the 4 or 5 flights they had lined up. Travelling though Stansted on a weekday early morning during summer was like a special type of freak show however - I've never seen as many neck/face/head tattoos or orange people with lip filler in all my life. Made Gatwick airport look like a walk around South Kensignton
Re: Computer says no
Posted: Mon Jul 22, 2024 9:19 am
by dinny_g
GG. wrote: Mon Jul 22, 2024 9:16 am
Travelling though Stansted on a weekday early morning during summer was like a special type of freak show however - I've never seen as many neck/face/head tattoos or orange people with lip filler in all my life. Made Gatwick airport look like a walk around South Kensignton
Re: Computer says no
Posted: Mon Jul 22, 2024 11:48 am
by Matty
Simon wrote: Sun Jul 21, 2024 9:51 pm
Also staggering that their CEO forgot to apologise on TV.
Have you seen what happened in Mr Robot Season 1 when a CEO apologised live on air for an IT outbreak? Probably for the best.
Basically, nope, wasn't a definition file but a separate update channel that's not user controllable, for sensor config - and their release management/QA system failed to catch the bad update as it went out.
Apparently they're gonna add user controllability to that (IE like they have for definition deployment), and - heaven forfend - release notes.
So er, that's not gonna fill many people with confidence - apparently their release management process either doesn't include much in the way of "spin up a Windows box, install Crowdstrike Falcon, apply latest update, see if it reboots OK", or it failed entirely, and you'd think testing of that process to make sure it's not gonna cause millions of dollars of damage if it goes wrong would be kiiiiinda important in a piece of software that operates in KernelMode/Ring0/as a god.
Re: Computer says no
Posted: Thu Jul 25, 2024 9:02 am
by Mito Man
Have you seen their apology? A $10 Uber Eats gift card
Re: Computer says no
Posted: Thu Jul 25, 2024 9:40 am
by Simon
Mito Man wrote: Thu Jul 25, 2024 9:02 am
Have you seen their apology? A $10 Uber Eats gift card
And lots of reports of the code not working....
Re: Computer says no
Posted: Thu Jul 25, 2024 11:32 am
by Beany
Cyber security company sends out what looks suspiciously like a phishing email.
