Whatsapp down

[Explosive Newt]

First petrol, now this.

[duncs500]

... and FB as far as I can tell. Hacked?

[240PP]

And FB/Insta. DNS issue apparently. Quite the fuck up somewhere.

[Broccers]

Trump revenge

[Explosive Newt]

I'm going to ask all the people I usually message via whatsapp to join ov9 so we can carry on the group chat.

[Beany]

And FB/Insta. DNS issue apparently. Quite the fuck up somewhere.

They'll be using dns for vast swathes of their platforms - insta, FB, WhatsApp - internally. Likely that that's crapped itself, starting small and cascading outwards.

Also issues with EE internet at the moment it seems, although that might just be local to me - had to connect to my home internet via a VPN to get DNS resolution working while on mobile data.

And I had a fairly serious database whooopsy at work today.

It's not a day for technology it seems.

[JonMad]

Clearly not my broadband playing up then. Good-o.

[240PP]

Also issues with EE internet at the moment it seems, although that might just be local to me

My EE 4G in Wiltshire is down now too FFS. The app is borked too.

[John]

We use EE at work and it seems to have more than its fair share of problems compared to other networks.

[Beany]

EE had been pretty reliable for me over the years up here in the grim North, so it took my by surprise that it was playing up

Anyway, latest rumours are a bad BGP push caused by poor automation/interface design that basically took Facebook's entire DNS infrastructure off the net

[nuttinnew]

Poor bunnies Let's hope Channing Tatum can come to the rescue.

[Explosive Newt]

EE had been pretty reliable for me over the years up here in the grim North, so it took my by surprise that it was playing up

Anyway, latest rumours are a bad BGP push caused by poor automation/interface design that basically took Facebook's entire DNS infrastructure off the net

Why/how would this happen?

[Beany]

Poor scoping of the tool chain, bad training, to many assumptions made about the ability to undo a bad change, etc.

Basically, human error by six degrees of separation.

[Beany]

https://krebsonsecurity.com/2021/10/wha ... -whatsapp/

Brian Krebs is a pretty big deal in the infosec world, although even he doesn't have any special insight, really.

Seems far more likely that someone made one of those changes to the platform that has had 'interesting' results, rather than anything malicious.

(I had one of those myself, today - who knew a MySQL analysis tool would dump gigabytes of data to your temporary folder? Not a problem of you have gobs of disk space free. I didn't....)

[Mito Man]

I was wondering why it was so quiet on Workplace today. I’m a fan of this

[DaveE]

"Hanlon's razor is a saying that reads: Never attribute to malice that which is adequately explained by stupidity."

[Beany]

"Hanlon's razor is a saying that reads: Never attribute to malice that which is adequately explained by stupidity."

Exactly this.

With infrastructure the scope of FBs, and the gobsmacking arrogance of their tech teams, then half assing a BGP change tool is far, far more likely than LOL MASSIVE HACK.

It's a non-zero possibility that there has been an external factor, but that non-zero value is extremely low compared to the hubris of people who think they run the world.

Edit: If you want to get into 'realistic conspiracy' land, a senior network tech who is disgruntled with the alleged political position FB takes (IE letting white supremacy and anti-vaxx shit get eyes on because ad revenue, regardless of consequences) has let a bad update go through, to ensure that FB stays in the news and eyes stay on the current controversy and whistleblowing.

In the same way that 'it's never DNS, it can't be DNS - it was DNS'..."it can't be someone inside who hates is, it's never someone inside who hates us.....it was someone on the inside who hates us"

[Ascender]

"Hanlon's razor is a saying that reads: Never attribute to malice that which is adequately explained by stupidity."

Yip, there's some info on Twitter which seems to suggests someone's f*cked up a config change which has taken their BGP peers down. Of course, they've somehow managed to do it on their resilient legs as well.* And then there's people saying they don't have any OOB access to their core devices either. Which sounds even more unbelievable for a company of that size.



* I'm sure a lot of us have these sort of stories, but the best one I was involved in first hand was when an engineer made a change for a large Bank which didn't work, but he didn't know why. So he did it on the other data centre network too, to see what happened. He brought down the entire bank's ATM network. Good times.

[Beany]

[Explosive Newt]

Rumours are this has severely ballsed up fb’s whole infrastructure and people’s ID cards won’t let them into the datacentres. Combined with reduced staffing owing to the pandemic, it doesn’t sound good.

* This is a global outage for all FB-related services/infra (source: I'm currently on the recovery/investigation team).

* Will try to provide any important/interesting bits as I see them. There is a ton of stuff flying around right now and like 7 separate discussion channels and video calls.

* Update 1440 UTC: \

As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).

There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.

Part of this is also due to lower staffing in data centers due to pandemic measures.

https://news.ycombinator.com/item?id=28748203
An interesting read