Going to think of this when I have a bad day at work. Can't imagine what it's like when it dawn's on you what just happened.

[removed]

Homer voice: 🎶Guess I forgot to put the fog lights in🎶

If it takes only one working person to throw an entire system into chaos like this then the process their following needs to be updated. Quickly.

At least in my worst fat finger fuck up ive never crippled an entire countrys air travel

Absolutely. Was there a pop up just saying "Are you absolutely fucking sure you want to do this"?

I feel like this is a scapegoat answer. Someone high up made a bad decision that led to a critical outage, methinks

Killall -9 contractors

More than that. There needs to be authorization to do such an action in the program and the process

[removed]

Not a software engineer but has that ring to it. They said something in the article along making the system more resilient, aka how the fuck could this happen?

Probably going to be a lot of pointing fingers up above. Would imagine this environment is pretty niche too.

Old guys rule. Some genius replaced an old guy with a contractor who was cheaper.

I think it's actually an accounting error that caused the problem. Too many damn accountants running the companies these days.

I legit thought there was another outage for a second there.

[removed]

[removed]

so did I

I'm a contractor in a completely different part of the government. If something goes wrong, they blame a contractor, any contractor it doesn't matter. I've sat in meetings where people blamed things on "a contractor from XYZ." Never a name, just "a contractor." Makes a person not care if their work shuts down all of US airspace or not.

Yep that matches up with my experience too

"this is written in fortran, let me fix that and make it javascript" - some contractor.

All it takes is a neo-Nazi with a gun to destroy a power station and plunge people into darkness.

I worked on an FAA contract in the 90’s. The thing that amazes me most is that stuff like this doesn’t happen regularly. It was so bad I actually resigned. A year later I hear the entire contract had to be redone. Rampant incompetence.

Shooting a power distribution station and cutting power to a town is a little bit different than shutting down all united states air travel

We added a resiliency power point slide.

>Two power substations in a North Carolina county were damaged by gunfire in what is being investigated as a criminal act, causing damage that could take days to repair and leaving tens of thousands of people without electricity, authorities said Sunday. Source

>About 35,000 people in North Carolina's Moore County remain without power on Wednesday after the substations were damaged in what authorities described as a "targeted" attack Source

> FBI warns of neo-Nazi plots as attacks on Northwest power grid spike Source

A simple Google search will show a disturbing pattern of attacks on power substations by suspected neo-Nazis. So these are not attacks by teenagers shooting up signs. These are attacks on the vulnerable infrastructure of the United States

[removed]

yes and now they know that.
Same with now people know that an individual person can bring a decades old system to its knees.

You must not work with users often.

In the case of the accidental Hawaii missile alert, it was very poor UI to have basically hyperlinks almost beside each other for "drill" and "not a drill". But then

>Still, there is a second confirmation page as a safety measure, asking if the employee is sure they want to send the alert, which they also mistakenly pressed “yes” on.

I once had a user use their Outlook trash bin as their filing cabinet. Then she went to "file" some messages from her inbox without realizing she was already in the trash. Outlook pops up a warning saying that the messages will be irrecoverably deleted, she press "yes", then when she called me in a panic and I said "the computer did exactly what you told it to do", she told me "well what are you guys even here for"

[removed]

[deleted]

>You must not work with users often.

I'll bungle this as my eyes are half open.

Yep, very important when writing the UAT scripts/procedure, at some points, to place yourself in the position of "brand new user, knows nothing." Also make sure your SME, if they're involved in the actual testing, remembers the same.

It would take many times the amount of time, people, and equipment that the NC attacks were perpetrated with in order to effect the kind of damage that the FAA problems created.

And they are not power stations, they’re distribution stations. Try shooting up a coal plant and see how far you get. Might even make it to within eyesight of it.

In the the most recent investment bank I worked at, no single person had the authority to make a change to a production system without multiple approvals, and a change would have to have a fellow technician verify the install and fallback process, there had to be documentation showing a test of the change as well as the implementation and fallback process on a staging system. If it was an emergency change that had implications to a live system with user impact, there would be 15 - 20 people dialed into a call and online to the system, observing every part of the change. Every user who would be impacted would have to be aware, and their C-level management would have to sign off on the change. And just getting the access to implement a one-time change required CIO approval.

But that's people's money. It's not just *actual lives at stake-*level stuff.

[removed]

"you ever hear of turbo pascal?"

[removed]

I'd say it's probably written in Jovial if we really want to go old school.