Do you have adblock enabled?
 
If you can read this, either the style sheet didn't load or you have an older browser that doesn't support style sheets. Try clearing your browser cache and refreshing the page.

(Wired)   Amazon engineer states that the NFL could have avoided the Super Bowl outage. If anyone knows anything about outages, it's Amazon   (wired.com ) divider line
    More: Obvious, Super Bowl, NFL, Amazon, Mercedes-Benz Superdome, SMG  
•       •       •

1181 clicks; posted to Sports » on 07 Feb 2013 at 11:11 AM (3 years ago)   |   Favorite    |   share:  Share on Twitter share via Email Share on Facebook   more»



47 Comments     (+0 »)
 
View Voting Results: Smartest and Funniest
 
2013-02-07 08:35:39 AM  
Seems pretty obvious that whomever was in charge of planning fumbled the ball.
 
2013-02-07 08:47:13 AM  
Hate to disagree with an Amazon Distinguished Engineer, but I am not sure I understand.

From other articles, I have understood that the utility has been screaming that they never interrupted the flow of power to the Superdome. Their feeder lines were always energized. The issue was within the distribution system of the Superdome itself.

If that is true, then I am not sure how diesel generators would help. If the problem is in the Superdome power distribution, then swapping out one working supply of electricity (the utility) with another (the generators) doesn't seem to be much help.

Or did I miss something?
 
2013-02-07 09:02:37 AM  

mr_a: Hate to disagree with an Amazon Distinguished Engineer, but I am not sure I understand.

From other articles, I have understood that the utility has been screaming that they never interrupted the flow of power to the Superdome. Their feeder lines were always energized. The issue was within the distribution system of the Superdome itself.

If that is true, then I am not sure how diesel generators would help. If the problem is in the Superdome power distribution, then swapping out one working supply of electricity (the utility) with another (the generators) doesn't seem to be much help.

Or did I miss something?


The lack of a UPS.  If something was tripped, it should have flipped to a UPS.  There's no excuse for the power outage since there are multiple ways to handle the situation.
 
2013-02-07 09:14:00 AM  
For about $100K, you could give every fan a souvenir flashlight.
 
2013-02-07 09:35:06 AM  
Of course the outage could have been avoided.

All they had to do was not play the Super Bowl in New Orleans.
 
2013-02-07 09:49:45 AM  

slayer199: mr_a: Hate to disagree with an Amazon Distinguished Engineer, but I am not sure I understand.

From other articles, I have understood that the utility has been screaming that they never interrupted the flow of power to the Superdome. Their feeder lines were always energized. The issue was within the distribution system of the Superdome itself.

If that is true, then I am not sure how diesel generators would help. If the problem is in the Superdome power distribution, then swapping out one working supply of electricity (the utility) with another (the generators) doesn't seem to be much help.

Or did I miss something?

The lack of a UPS.  If something was tripped, it should have flipped to a UPS.  There's no excuse for the power outage since there are multiple ways to handle the situation.



It would be quite the UPS to handle the Superdome. I can't find a quoted number for the electricity draw of the Superdome, but it has got to be on the order of Megawatts. The largest UPS in the world, the BESS system in Alaska, is quoted as supplying 47MW for 15 minutes. The Superdome power outage was roughly an hour, so this is at least the rough order of magnitude system that would be needed.  The BESS system cost $35M, and takes up roughly a soccer field-a bit of an issue for downtown New Orleans.

As it was, the Superdome stayed lit, at least enough to prevent panic and injury. Not sure if that was because of a UPS, or just that the place seems to have two separate distribution systems, only one of which was effected.

http://library.abb.com/global/scot/scot271.nsf/veritydisplay/627968b e8 161966fc1256e3f004e0366/$File/38-43%20M848.pdf

http://www.wired.com/science/discoveries/news/2008/08/dayintech_0827
 
2013-02-07 10:00:30 AM  

mr_a: It would be quite the UPS to handle the Superdome. I can't find a quoted number for the electricity draw of the Superdome, but it has got to be on the order of Megawatts. The largest UPS in the world, the BESS system in Alaska, is quoted as supplying 47MW for 15 minutes. The Superdome power outage was roughly an hour, so this is at least the rough order of magnitude system that would be needed. The BESS system cost $35M, and takes up roughly a soccer field-a bit of an issue for downtown New Orleans.

As it was, the Superdome stayed lit, at least enough to prevent panic and injury. Not sure if that was because of a UPS, or just that the place seems to have two separate distribution systems, only one of which was effected.


You use a UPS to keep the power going so there's no interruptions, and generators to supply power if the breakers are tripped.  A UPS of that scale isn't meant to supply power to the entire building for hours...just long enough for generators to fire up and maintain power.  That was the point of TFA.
 
2013-02-07 10:13:22 AM  

slayer199: mr_a: It would be quite the UPS to handle the Superdome. I can't find a quoted number for the electricity draw of the Superdome, but it has got to be on the order of Megawatts. The largest UPS in the world, the BESS system in Alaska, is quoted as supplying 47MW for 15 minutes. The Superdome power outage was roughly an hour, so this is at least the rough order of magnitude system that would be needed. The BESS system cost $35M, and takes up roughly a soccer field-a bit of an issue for downtown New Orleans.

As it was, the Superdome stayed lit, at least enough to prevent panic and injury. Not sure if that was because of a UPS, or just that the place seems to have two separate distribution systems, only one of which was effected.

You use a UPS to keep the power going so there's no interruptions, and generators to supply power if the breakers are tripped.  A UPS of that scale isn't meant to supply power to the entire building for hours...just long enough for generators to fire up and maintain power.  That was the point of TFA.


You're still talking about a gigantic battery to keep things going for even 10 seconds to get generators going. The energy use in the Superdome had to be enormous.
 
2013-02-07 10:47:39 AM  

GAT_00: slayer199: mr_a: It would be quite the UPS to handle the Superdome. I can't find a quoted number for the electricity draw of the Superdome, but it has got to be on the order of Megawatts. The largest UPS in the world, the BESS system in Alaska, is quoted as supplying 47MW for 15 minutes. The Superdome power outage was roughly an hour, so this is at least the rough order of magnitude system that would be needed. The BESS system cost $35M, and takes up roughly a soccer field-a bit of an issue for downtown New Orleans.

As it was, the Superdome stayed lit, at least enough to prevent panic and injury. Not sure if that was because of a UPS, or just that the place seems to have two separate distribution systems, only one of which was effected.

You use a UPS to keep the power going so there's no interruptions, and generators to supply power if the breakers are tripped.  A UPS of that scale isn't meant to supply power to the entire building for hours...just long enough for generators to fire up and maintain power.  That was the point of TFA.

You're still talking about a gigantic battery to keep things going for even 10 seconds to get generators going. The energy use in the Superdome had to be enormous.


You're not talking about a significant amount of time. Small Datacenters usually do this in minutes (under 5). Our DC is on generator in under 5 minutes. It could be less but it's designed not to fire up the generator unless there's a true loss of power and not just a hiccup.
 
2013-02-07 11:11:24 AM  

slayer199: mr_a: It would be quite the UPS to handle the Superdome. I can't find a quoted number for the electricity draw of the Superdome, but it has got to be on the order of Megawatts. The largest UPS in the world, the BESS system in Alaska, is quoted as supplying 47MW for 15 minutes. The Superdome power outage was roughly an hour, so this is at least the rough order of magnitude system that would be needed. The BESS system cost $35M, and takes up roughly a soccer field-a bit of an issue for downtown New Orleans.

As it was, the Superdome stayed lit, at least enough to prevent panic and injury. Not sure if that was because of a UPS, or just that the place seems to have two separate distribution systems, only one of which was effected.

You use a UPS to keep the power going so there's no interruptions, and generators to supply power if the breakers are tripped.  A UPS of that scale isn't meant to supply power to the entire building for hours...just long enough for generators to fire up and maintain power.  That was the point of TFA.


OK, that takes me back to my original thought.

Maybe we just have different ideas on what the Superdome power system looks like. I envision a single point (lets call it point A) where the utility feeds cross the fence to the Superdome. At that point, there is a small substation consisting of a couple transformers and probably some big caps to balance the load factor. From there, there are various high-voltage feeder lines running to various points in the Superdome, where yet more transformers further step down the voltage and probably adjust the load. Also buried in this network are assorted relays, circuit breakers, framistats, and who knows what else.

If we take the local utility at its word, it never stopped delivering power to point A, so the problem was somewhere between point A and the light bulbs. Since there have also been reports of quite a bit of "emergency work" done within the Superdome's electrical system last month, this seems to make sense.

The problem is that any other source of power, be it a UPS, generator or even a different utility feed most likely also connects at point A, although it may skip a transformer or two. So I am still not sure I understand what value a UPS or generator bank would have provided.
 
2013-02-07 11:13:35 AM  
Why do people care? Some people had to sit in the dark for about an hour. It's not even like it was the 4th quarter and they'd cut off beer sales.
 
2013-02-07 11:17:40 AM  
Sorry, when the Illuminati wants the power to go out, so they can conduct a ritual at the biggest televised event in the world, the power goes out!

Don't believe me, just ask Alex Jones.   He's onto Beyonce and her Illuminati hand signals!
 
2013-02-07 11:19:17 AM  

lawboy87: Sorry, when the Illuminati wants the power to go out, so they can conduct a ritual at the biggest televised event in the world, the power goes out!

Don't believe me, just ask Alex Jones.   He's onto Beyonce and her Illuminati hand signals!


The Illuminati tweeted that it wasn't them, though.
 
2013-02-07 11:22:31 AM  
I've heard conspiracy theories that it was Vegas bookmakers trying to interrupt Baltimore's momentum, but I think it's much more plausible that it was New Orleans saying "Fark You" to Goodell.
 
2013-02-07 11:23:01 AM  

IAmRight: Why do people care? Some people had to sit in the dark for about an hour. It's not even like it was the 4th quarter and they'd cut off beer sales.


Because someone has to answer for this outrage and heads have to roll.  Now if you ask why does someone have to answer and why do heads have to roll, that is a deeper question.  My guess would be several someones' egos are ruffled and will only be soothed by making other people suffer.

It's also a great way for people who toil in relative anonymity to get their 15 minutes by offering plausible armchair advice without taking any responsibility for their words.  When else are you going to feel like a sports broadcaster/pregame panelist?
 
2013-02-07 11:27:03 AM  
This is assuming the fault was with external power supply, and not a short within the internal electrical system of the dome.
 
2013-02-07 11:27:41 AM  

mr_a: It would be quite the UPS to handle the Superdome.


Also, It wouldn't necessarily need to supply power to everything in the stadium.  it could just supply power to things that take forever to warm up like the stadium lights.
 
rka
2013-02-07 11:29:08 AM  

Lost Thought 00: This is assuming the fault was with external power supply, and not a short within the internal electrical system of the dome.


Well, if you can't make wild-ass assumptions with absolutely no facts on the internet what's the point?
 
2013-02-07 11:29:48 AM  
Throw in a third generator, and the entire price tag would come to about $10 million, Hamilton estimated. He ought to know. He's the guy who figures out how to keep Amazon's dozens of data centers from going dark.

That $10 million is not much money compared to the 9.5 billion the NFL brought in last year.


Why don't you worry about your own shiat?

http://www.theregister.co.uk/2012/10/22/amazon_east_coast_down_again /

Another set of backup servers at every Amazon data site would only cost around $100 million.  That $100 million is not much money compared to the $61 billion Amazon brought in last year.

How about you run it past you boss and see if he'll appropriate some money for it?  I'm sure the shareholders won't mind the profit loss.

Wow, what an insufferable hypocrite?  "Posted on his blog" should have been your first clue.  Fail all around here.
 
2013-02-07 11:34:48 AM  

IAmRight: Why do people care? Some people had to sit in the dark for about an hour.


You're aware they were sharing the dark with Ray Lewis?
 
2013-02-07 11:37:15 AM  
1. Extend game 34 minutes by shutting off lights
2. Show more commercials
3. Profit
 
2013-02-07 11:38:04 AM  
I've used UPSes (UPSs?) in broadcast situations and have twice been in situations where after an outage the UPS itself also fails. Bringing in more failsafes always helps, but at a certain point, if Zeus wants you to lose power, you're just going to lose power.
 
2013-02-07 11:39:22 AM  

Mike_LowELL: You're aware they were sharing the dark with Ray Lewis?


I'd be more worried about 49ers fans - they've stabbed several people this year.
 
2013-02-07 11:40:53 AM  
BTW, I really don't mind if his crew did stab those people to death. If you start a bar fight and break a bottle over someone's head, you get to deal with whatever comes at you, and if you die, the world ain't missing out on much.
 
2013-02-07 11:44:03 AM  

IAmRight: BTW, I really don't mind if his crew did stab those people to death. If you start a bar fight and break a bottle over someone's head, you get to deal with whatever comes at you, and if you die, the world ain't missing out on much.


*popcorn*
 
2013-02-07 11:49:51 AM  
It's been fine for 40 something years and one mishap makes everyone go apesh*t?  Of course they could have had backup generators but what about all the other unexpected possibilities?  The toilets go out, well it wouldn't have been a problem if they had 5000 port-a-pottys on stand-by just in case.  Terrorist attack?  Well, it wouldn't have been a problem if they had 50,000 troops guarding the stadium.
 
2013-02-07 12:04:36 PM  
why hasnt the superdome been replaced?
 
2013-02-07 12:10:10 PM  
The outage could have been avoided if the Batman had been in the Dome.
 
2013-02-07 12:13:36 PM  
It doesn't matter where the power is coming from. If you don't isolate the fault before connecting another source of power, then the generator's or UPS' breaker would have tripped as well. Plus the lights take 15 minutes to come up once they have power. Can't say much more without knowing exactly what in the stadium drew the surge.
 
2013-02-07 12:28:12 PM  

Rapmaster2000: Wow, what an insufferable hypocrite? "Posted on his blog" should have been your first clue. Fail all around here.


I do not believe the recent AWS outage was power related so you go yell at the wind some more.
 
2013-02-07 12:34:57 PM  

JohnBigBootay: Rapmaster2000: Wow, what an insufferable hypocrite? "Posted on his blog" should have been your first clue. Fail all around here.

I do not believe the recent AWS outage was power related so you go yell at the wind some more.


I don't think it's a stretch to mock an armchair operations manager who can't successfully manage his own operations.  His "expertise" in critiquing the failures of others regarding outages and his own demonstrated failure leaves him ripe for ridicule.  It wouldn't matter if it was a network outage, a cooling outage, a performance management issue, or anything else relating to the management of an integrated technological system.

White knighting for farkups won't get you promoted from behind the help desk.
 
2013-02-07 12:42:51 PM  

Watubi: It's been fine for 40 something years and one mishap makes everyone go apesh*t?  Of course they could have had backup generators but what about all the other unexpected possibilities?  The toilets go out, well it wouldn't have been a problem if they had 5000 port-a-pottys on stand-by just in case.  Terrorist attack?  Well, it wouldn't have been a problem if they had 50,000 troops guarding the stadium.


Are you available to work on a Godzilla contingency plan?  Difficulty: No Mothra fail safes will be in place for the event.
 
2013-02-07 12:47:36 PM  

Rapmaster2000: I don't think it's a stretch to mock an armchair operations manager


http://mvdirona.com/jrh/work/  - yeah, nothing says "armchair" operations manager like this guy's resume. They probably have to keep the outlets covered up at his house so he doesn't unknowingly stick a fork in them.

But you're right, as always.  We don't need distinguished engineers exercising their first amendment rights in their personal blog and making knowledgeable, respectful commentary about an infrastructure event seen by hundreds of millions of people. He should shut his farking mouth because aws on the east coast doesn't have 100% up time.
 
2013-02-07 12:49:49 PM  
The Superdome does have backup generators.  Power was not out for 34 minutes.  I don't know the exact length of the outage, but it was out for less than 5 and probably less than 1.  It takes a few minutes to verify that the power is good, and then start to switch everything back on.  It takes that long to do a cold start on a facility the size of the Superdome.  And then those lights take 15+ minutes to warm up and come up to brightness.  And all the cameras and timekeeping equipment need to be checked and all the technology on the sidelines.  Now, depending on where in the system the fault occured, a UPS may or may not have been beneficial.

In conclusion, this is much ado about nothing.  If you look at the cost of providing 100% uptime on a facility that's only in use for a few hours a week, I doubt that it's worth it.  Sometimes it's just a risk that you have to accept.
 
2013-02-07 01:01:27 PM  

JohnBigBootay: Rapmaster2000: I don't think it's a stretch to mock an armchair operations manager

http://mvdirona.com/jrh/work/  - yeah, nothing says "armchair" operations manager like this guy's resume. They probably have to keep the outlets covered up at his house so he doesn't unknowingly stick a fork in them.

But you're right, as always.  We don't need distinguished engineers exercising their first amendment rights in their personal blog and making knowledgeable, respectful commentary about an infrastructure event seen by hundreds of millions of people. He should shut his farking mouth because aws on the east coast doesn't have 100% up time.


Distinguished is actually his title.  They're called "Senior Staff Engineers" where I work.  I'm only a notch below at Staff.

I find it pretty Duning-Krueger of him to chime right in with an "easy" solution to a system he has no experience with.  Not only that, but to come up with some ballpark figure when he knows that pitching an equivalent percentage ballpark figure at his boss for backups will get his boss asking why he needs backups in the first place.

He should know better than to act this way.  It's not a question of how brilliant of an engineer he is.  It's a question of how arrogant of an individual he is.

I've got experience here.  I'm on the phone with a Distinguished Engineer at a larger technology corporation than Amazon every week and I get this from him all the time.  It's amazing what a title can do to someone's ability to self-evaluate competence.

Full disclosure:  I have nothing against Amazon. My cousin is a VP there.
 
2013-02-07 01:24:23 PM  

NewWorldDan: Now, depending on where in the system the fault occured, a UPS may or may not have been beneficial.


It's a matter of design. The point is that proper design and planning would have made the outage largely preventable.  Datacenters all over the world are built this way...and yes, it isn't perfect, but it works 99.99% of the time.

Here are a few examples:

classroomrevolution.com
www.alphagrissin.bg
www.alphagrissin.bg
 
2013-02-07 01:46:04 PM  
JohnBigBootay:

http://mvdirona.com/jrh/work/  - yeah, nothing says "armchair" operations manager like this guy's resume. They probably have to keep the outlets covered up at his house so he doesn't unknowingly stick a fork in them.

But you're right, as always.  We don't need distinguished engineers exercising their first amendment rights in their personal blog and making knowledgeable, respectful commentary about an infrastructure event seen by hundreds of millions of people. He should shut his farking mouth because aws on the east coast doesn't have 100% up time.


I just hate to see guys think that executive hair makes them infallible.
 
2013-02-07 02:00:35 PM  

you have pee hands: I just hate to see guys think that executive hair makes them infallible.


I don't get why people are on this guy. He doesn't come off as infallible know it all guy to me in the least. He was an auto mechanic who worked his way all the way up to Fancy Engineer fer chrissakes. And he wasn't a jerk about it in the least. He has tons of relevant experience and he frequently posts on power failures, redundancy, and large infrastructure topics all the time. I guess this will learn him not to broach topics with some sort of connection to football.
 
2013-02-07 02:03:58 PM  

JohnBigBootay: I don't get why people are on this guy. He doesn't come off as infallible know it all guy to me in the least. He was an auto mechanic who worked his way all the way up to Fancy Engineer fer chrissakes. And he wasn't a jerk about it in the least. He has tons of relevant experience and he frequently posts on power failures, redundancy, and large infrastructure topics all the time. I guess this will learn him not to broach topics with some sort of connection to football.


Sorry, it was a weak joke about the picture with his resume.
 
2013-02-07 02:06:24 PM  

you have pee hands: Sorry, it was a weak joke about the picture with his resume.


No worries. His hair just screams CANADA to me. I don;t know why. i think it gives me a little wayne's world vibe.
 
2013-02-07 02:08:12 PM  

JohnBigBootay: you have pee hands: I just hate to see guys think that executive hair makes them infallible.

I don't get why people are on this guy. He doesn't come off as infallible know it all guy to me in the least. He was an auto mechanic who worked his way all the way up to Fancy Engineer fer chrissakes. And he wasn't a jerk about it in the least. He has tons of relevant experience and he frequently posts on power failures, redundancy, and large infrastructure topics all the time. I guess this will learn him not to broach topics with some sort of connection to football.


Alright, well I didn't read his blog.  I apologize if he was contrite.  The wired article didn't do him any favors.

I'm probably still touchy from reading Fark's Solutions to the Deepwater Horizon Leak aka "How come they never thought to put like a huge pile of sandbags on it?"
 
2013-02-07 03:56:12 PM  

Rapmaster2000: JohnBigBootay: you have pee hands: I just hate to see guys think that executive hair makes them infallible.

I don't get why people are on this guy. He doesn't come off as infallible know it all guy to me in the least. He was an auto mechanic who worked his way all the way up to Fancy Engineer fer chrissakes. And he wasn't a jerk about it in the least. He has tons of relevant experience and he frequently posts on power failures, redundancy, and large infrastructure topics all the time. I guess this will learn him not to broach topics with some sort of connection to football.

Alright, well I didn't read his blog.  I apologize if he was contrite.  The wired article didn't do him any favors.

I'm probably still touchy from reading Fark's Solutions to the Deepwater Horizon Leak aka "How come they never thought to put like a huge pile of sandbags on it?"


A huge pile of sandbags probably wouldn't have helped the Superbowl-unless they dropped them on Phil Simms.
 
2013-02-07 04:46:45 PM  

Rapmaster2000: Alright, well I didn't read his blog. I apologize if he was contrite. The wired article didn't do him any favors.


Yeah, they have to make it sound all accusatory and inflammatory. I don't think he criticized a single soul. My guess is you'd enjoy his blog. Why I read shiat like that when the content is usually waaaaay over my head is a mystery to me.
 
2013-02-07 05:14:29 PM  
At my place of employment they recently made some changes in one of the restaurants, adding new equipment.   Shortly after the new equipment went online, a dishwasher threw a ground fault.  Taking out circuits all the way back to the main.  Pitch black for twenty minutes while the facilities guys hunted down the problem.  We've got a generator and UPS on the computer systems.  But for 20 minutes it was damn dark because the generator didn't kick in.  Why?  Because it never saw a loss of utility power.  They couldn't bring power up by manually kicking on the generator because of where the circuits were down.

Seems to me this was a similar situation.  The engineers have to find out where the problem is before they can start powering shiat up again because otherwise it will just blow again and cause more problems.

It's easy to Monday morning quarterback something like this.  But without being on site, with your hands on the equipment it's impossible to say how it could have been prevented.
 
2013-02-07 06:43:47 PM  
AmazOWNED
 
2013-02-08 10:06:18 AM  

Dull Cow Eyes: 1. Extend game 34 minutes by shutting off lights
2. Show more commercials
3. Profit


Except they didn't. That's why you had half an hour of useless field shots and ceaseless blather from the 2 sideline reporters who were miced up until they could swap out to an external commentary crew.
 
2013-02-08 03:47:40 PM  
Hard to believe the networks don't have some sort of "flying standby" equivalent for commercials during sporting events.
 
Displayed 47 of 47 comments

View Voting Results: Smartest and Funniest

This thread is archived, and closed to new comments.

Continue Farking
Submit a Link »
On Twitter








In Other Media
  1. Links are submitted by members of the Fark community.

  2. When community members submit a link, they also write a custom headline for the story.

  3. Other Farkers comment on the links. This is the number of comments. Click here to read them.

  4. Click here to submit a link.

Report