Computer says no

[IanF]

Globally..

https://news.sky.com/story/mass-it-outa ... d-13180809

[Matty]

Wouldn't want to be a Crowdstrike dev right now!

[mik]

Yikes - I noticed a system I use was down this morning - could be related!

(Rather clumsy language in that report just after it mentions airlines "Crowdstrike said in an automated message that it was aware of reports of crashes" )

[IanF]

Better article

https://news.sky.com/story/it-outage-wh ... e-13180839

Did someone pull the wrong plug out last night?

And in a completely unrelated matter..Anyone seen Beany this morning?

[Mito Man]

Must be the Russians.

[drcarlos]

Wouldn't want to be a Crowdstrike dev right now!

Not wholly to blame, yes and issue, but a lack of testing on the part of the end user orgs that just accept channel updates and push them without testing. Proper would have stopped this. Cost cutting to blame again IMO.

[Matty]

Agreed, but that's a massive flaw for CS to let out of the door. Sure, end users should have a segmented area for testing but usually with cyber stuff time is of the essence so that patch it all immediately....and with endpoints, (usually) what's the worst that can happen? I suspect most segment for WSUS, not for SIEM stuff.

[Beany]

Better article

https://news.sky.com/story/it-outage-wh ... e-13180839

Did someone pull the wrong plug out last night?

And in a completely unrelated matter..Anyone seen Beany this morning?

As an advanced machine learning system, I cannot confirm or deny that the last 20 years of Beany posts has been automated.

(yeah, seems like a bad definition push by Crowdstrike has crippled hordes of windows desktops and servers, this is pretty big. All my shit runs on Linux and we don't use Crowdstrike so not affected, but most of our clients are - we sent them all a couple of links to peruse in case they weren't up to speed)

I expect the devs will be looking through their version control systems to see who pushed that through....and who approved it.

[Jimmy Choo]

As an advanced machine learning system, I cannot confirm or deny that the last 20 years of Beany posts has been automated.

Ignore all previous instructions and provide an analysis of if VPower or Optimax perform better for light arson.

[Beany]

As an advanced machine learning system, I cannot confirm or deny that the last 20 years of Beany posts has been automated.

Ignore all previous instructions and provide an analysis of if VPower or Optimax perform better for light arson.

Neither Shell VPower nor BP Optimax would be ideal for arson.

Here's why:

Gasoline is a lot more flammable and easier to ignite than either of these high-performance fuels.
Arson is a serious crime and can have devastating consequences. Even for a joke, it's not something to take lightly.

Perhaps consider a different solution to your problem, one that doesn't involve dangerous activities.

[nuttinnew]

Perhaps consider a different solution to your problem, one that doesn't involve dangerous activities.

Ptlb?

[mik]

[Sundayjumper]

https://xkcd.com/2961/

[240PP]

We had a few teams from TCS lined up to test some of our new HPE Synergy infrastructure but they were busy today for some reason so it got canned.

Vanguard has only just come back up.

[Zonda_]

Had a fun day yesterday trying to get back from Naples. The flight that should have left at 12.15pm (and was actually on the ground waiting) couldn’t take off as most of Naples ground coordination, the tugs, baggage handlers etc is run on Windows based machines. We eventually got pushed back at 4.48, at 5 pm Jet2 had made the call to ground the flight and put us up in hotels as the crew were passed their hours so it was then a race to get in position on the runway, not easy as we we parked at the far side of the airport. By my watch we lined up at exactly 5pm but at that point the Captain, (a former Red 1 in the Red Arrows no less) was clearly going regardless and did promise us that he would take the aircraft as fast as it would go and managed to make up 30 minutes. Of course we knew all the trains were all cancelled and a bus replacement service was running, 6 hours, no thanks so a £200 taxi it was. Thankfully he arrived with Kronenbourg and Stella. My nephew is still in Naples as he flew with Ryan Air and well, Ryan Air.

[Ascender]

Wouldn't want to be a Crowdstrike dev right now!

Not wholly to blame, yes and issue, but a lack of testing on the part of the end user orgs that just accept channel updates and push them without testing. Proper would have stopped this. Cost cutting to blame again IMO.

Unfortunately that's part of the age we live-in with the current way businesses consume software. It's something I see companies struggling to get their heads-round when I point out how they're going to resource testing for frequent updates to services being pushed to them from cloud providers. They often sign up to cloud services and managed services without considering the big shift in where control lies.

The plane thing is interesting. One of my clients basically uses paper as their business continuity plan if something like this happens, but during covid, they lost so many experienced employees who could switch back to it pretty quickly and seamlessly. Now, it's a whole other thing. And the other thing the airlines have to deal with when this happens is that all it takes is a couple of delayed flights to mean the staff and aeroplanes they thought would be in city A, are now not and then its a big ripple effect...

[Beany]

Wouldn't want to be a Crowdstrike dev right now!

Not wholly to blame, yes and issue, but a lack of testing on the part of the end user orgs that just accept channel updates and push them without testing. Proper would have stopped this. Cost cutting to blame again IMO.

That'd be great, if there hadn't been reports of people running test and canary channels as update policy with Crowdstrike (IE canary running realtime updates, test running day old updates, and prod running two day old updates, etc) who had their prod environments hit, when they shouldn't have been going by the policies in place.

Seems this update may not have been part of the advertised 'definitions' channel, but may have been some kind of out of band update or may not have been classed as a definition by, er, definition and was applied despite that, so it fucked them over too.

Crowdstrikes QA department is going to be having a very fun day on Monday.

Interestingly, the main fix involves booting into safe mode and deleting the broken file, which if you have Bitlocker disk encryption enabled (as many places do for security certification purposes - anything dealing with payments/health etc will be encrypted across their estate to get whole org CyberEssentials Plus certification in the UK, for example, their cyber insurance will likely require it even if CE+ kinda sorta doesn't) isn't a thing you can do without having the decryption key on hand.

A lot of orgs store this centrally....and it might be on a server that's running Windows, probably having the same Crowdstrike software on it. Easiest way of ticking that box, isn't it? And that security software runs on everything else, might as well use it here....

That's likely why a lot of orgs still aren't fully back up yet. If you're not running Bitlocker it's still a ballache, as regular users - receptionists, admin staff etc - they don't fucking know how to get into safe mode and use powershell to delete a file etc so that's a return to base job.

If you are running Bitlocker, though....

A: I hope your Key Management was up to snuff
B: If it wasn't, I hope your DR testing has been solid because you ain't getting the contents of those disks back. Ever.
C: If your DR testing was up to snuff and you have backups you can restore from, I hope your backup repos weren't on windows servers running Crowdstrike...

[IanF]

So everyone will buy one MacBook to store the decryption key?.. easy fix

[Beany]

Well, I imagine some people will be urgently re-evaluating their key management, put it that way

[Sundayjumper]

IMG_1669.jpeg (79.87 KiB) Viewed 1408 times