If you can read this, either the style sheet didn't load or you have an older browser that doesn't support style sheets. Try clearing your browser cache and refreshing the page.

(MSNBC)   Monster DC storm takes down Amazon's cloud servers, knocking Pinterest, Instagram offline. And nothing of value was lost   (redtape.msnbc.msn.com) divider line 84
    More: Cool, Amazon, error messages, cloud server  
•       •       •

2817 clicks; posted to Geek » on 30 Jun 2012 at 6:06 AM (2 years ago)   |  Favorite    |   share:  Share on Twitter share via Email Share on Facebook   more»



84 Comments   (+0 »)
   
View Voting Results: Smartest and Funniest

Archived thread

First | « | 1 | 2 | » | Last | Show all
 
2012-06-30 03:22:32 AM
Go out and take a walk people.
 
2012-06-30 04:29:45 AM
Jesus, cretinbob is right. I love the internet and all, but these people have never seen sunlight. "Netflix isn't working on possibly the most emotional night of my life. It's official ... this is NOT real life," Are you farking kidding me?!

Wow.
 
2012-06-30 04:48:11 AM
Are Amazon quite bad? I thought these big hosting facilities had umpteen backups and redundancies in place, and this isn't the first time something has happened to knock Amazon using sites off.

But lol at the mass hysteria, how can a night be the most emotional of his life? I bet s/he's 14.
 
2012-06-30 06:17:07 AM
Hey, Woot was down, too. There was definitely value lost
 
2012-06-30 06:28:20 AM
Earth says, "I got your cloud right here baby".
 
2012-06-30 06:41:08 AM

Slaxl: Are Amazon quite bad? I thought these big hosting facilities had umpteen backups and redundancies in place, and this isn't the first time something has happened to knock Amazon using sites off.


They're normally quite good. It's just that most customers don't take advantage of their multi-availability-zone (or multi-region) fail-over capability because that costs money and people are lazy and cheap.

Hosting a server (or several servers) on AWS is not really any different than any other VPS, but they have a lot of features for monitoring systems, spinning up new instances as needed, replication and fail-over between regions, etc.

In regards to reliability, sometimes shiat happens. Mother nature is a biatch.
 
2012-06-30 06:50:29 AM
This cloud-on-cloud violence has to stop!
 
2012-06-30 06:55:06 AM
With Pinterest down, my Wife actually wanted to talk.
 
2012-06-30 07:01:32 AM

cretinbob: Go out and take a walk people.


Are you kidding? Walking in the storm? That's crazy talk!

Netflix wasn't an option anyway for my partner and me. Our power in Fairfax went out, so we've refugeed to the in-laws in Loudon.
 
2012-06-30 07:14:13 AM

heypete: Slaxl: Are Amazon quite bad? I thought these big hosting facilities had umpteen backups and redundancies in place, and this isn't the first time something has happened to knock Amazon using sites off.

They're normally quite good. It's just that most customers don't take advantage of their multi-availability-zone (or multi-region) fail-over capability because that costs money and people are lazy and cheap.

In regards to reliability, sometimes shiat happens. Mother nature is a biatch.


This was totally preventable. No datacenter should be at the whims of the grid...ever. Amazon didn't install enough fail-over capablilty in their power distribution or test it properly because that costs money and people are lazy and cheap.

Rackspace and Google don't seem to be having this recurring problem.
 
2012-06-30 07:14:50 AM

cretinbob: Go out and take a walk people.


The people in which country are most addicted to their electronic devices and social networks? C) USA! USA! USA!

We go to Europe and have our faces stuck in our iPhones most of the time.
 
2012-06-30 07:41:03 AM

IoSaturnalia: heypete: Slaxl: Are Amazon quite bad? I thought these big hosting facilities had umpteen backups and redundancies in place, and this isn't the first time something has happened to knock Amazon using sites off.

They're normally quite good. It's just that most customers don't take advantage of their multi-availability-zone (or multi-region) fail-over capability because that costs money and people are lazy and cheap.

In regards to reliability, sometimes shiat happens. Mother nature is a biatch.

This was totally preventable. No datacenter should be at the whims of the grid...ever. Amazon didn't install enough fail-over capablilty in their power distribution or test it properly because that costs money and people are lazy and cheap.

Rackspace and Google don't seem to be having this recurring problem.


Rackspace and Google have both had issues, tho not as widespread.

My last company used to use Rackspace for email and I saw some crazy shiat. My favorite was a handful of accounts that reverted to the previous day's contents then "rolled forward" until they caught up with the present. First, a day's worth of email that had been read and filed disappeared. Then the missing messages started showing up in the inbox...then started moving to the folders where they'd been filed. (BTW, Rackspace was using Amazon to store the mail system backups at the time.) We tended to have one or two noticeable service interruptions per year. Despite allegedly having instant fail-over to their other data center (at the time, they had two for email services), we lost access completely for an extended period when one of their data centers went down. Apparently the fail-over didn't take.

Last spring, Google wiped out around 40,000 gmail accounts. Proper wiped. It took them about 20 hours to restore from tape. Anything sent to those accounts while they were unavailable was lost.
 
2012-06-30 07:51:50 AM

cretinbob: Go out and take a walk people.


In a thunderstorm?
 
ZAZ [TotalFark]
2012-06-30 07:58:31 AM
I thought these big hosting facilities had umpteen backups and redundancies in place

Once upon a time New England was cut off from the net. The region was behind seven redundant connections. All of them were routed over the same fiber. http://catless.ncl.ac.uk/Risks/4.30.html#subj1
 
2012-06-30 08:00:00 AM

Infobahn: With Pinterest down, my Wife actually wanted to talk.


I'm so sorry.
 
2012-06-30 08:01:18 AM
I had to re-read the first half of the headline about three times before I realized they were talking about a storm in Washington DC and not some sort of comic event. Too early in the morning I guess.
 
2012-06-30 08:01:35 AM

cretinbob: Go out and take a walk people.


I was there man!

70-90mph winds, sheets of rain, lightning like a MoFo.

I tried to stand in the garage and smoke a cig, no dice brother. Rain was coming 10 feet inside. Haven't seen a storm like that since nineteen ought-dickety dick!!

Just watched a movie on the DVR instead. ;)
 
2012-06-30 08:06:49 AM
Yeah it was a rough one here last night in the DC area. Dominion Virginia Power says 450,000 of its 800,000 Northern Virginia customers lost power. Mine blinked but didn't go out. Buried power lines FTW!
 
2012-06-30 08:18:31 AM
The data centers I have been in have rooms full of batteries, Diesel generators, and multiple substations and are rated for F4 tornadoes. Unless they lost the building they should have been OK.
 
2012-06-30 08:18:57 AM
Sure needed the rain, not all at once mind you, but it was needed.

There goes small stream fishing for three days...
 
2012-06-30 08:27:07 AM
Netflix was out too. Friday nights without a chickflick to put the lady in the mood probably CB'd more than a few date nights. This should be a lesson to you young men of dating age: always have a backup plan, like chocolates, or back rubs.
 
2012-06-30 08:33:17 AM
monster storm?

yup.

that bastard blew threw So. PA... winds were up to 70. Fricken nuts
 
2012-06-30 08:40:37 AM
The DC was lost, nothing anyone could do besides wait.
 
2012-06-30 09:02:38 AM

Walker: Yeah it was a rough one here last night in the DC area. Dominion Virginia Power says 450,000 of its 800,000 Northern Virginia customers lost power. Mine blinked but didn't go out. Buried power lines FTW!


Was picking my son up when that storm hit. From a dead (hot, muggy) calm the wind came up so fast and from so many different directions that I had a real fear we were inside a developing tornado, so I drove like hell home through sheeting rain and watched huge tree limbs dropping like flies all around me, actually saw grids losing power as i drove through them.


Seen a LOT of bad storms in my life, including ones in the high desert that featured hurricane-force winds, but I can't remember one that became SO violent so incredibly quickly
 
2012-06-30 09:03:48 AM
this is like the 3rd or 4th time in the past two years netflix. get your sh*t sorted out.
 
2012-06-30 09:07:36 AM
Damn big storm. I was flying to Columbus, and we routed through Charlotte(still made it to Columbus). It's was a nasty nasty storm viewed from the air.

Currently stuck in a hotel in BFE, because the place and reason I was supposed to be working at is without power.

At least the hotel has AC.
 
2012-06-30 09:10:18 AM
Yay another outsourcing gone wrong thread! What is Drew laying off the fark IT staff or something?
 
2012-06-30 09:30:45 AM
The weather is farked nation-wide right now; I'm surprised we haven't seen a thread about it yet. Record breaking high temperatures, heat-related mortality threats through the southeast, and the inter-mountain west is on fire.

i46.tinypic.com
 
2012-06-30 09:35:13 AM

LouDobbsAwaaaay: The weather is farked nation-wide right now; I'm surprised we haven't seen a thread about it yet. Record breaking high temperatures, heat-related mortality threats through the southeast, and the inter-mountain west is on fire.

[i46.tinypic.com image 640x640]


Where's all the skeptics?
 
2012-06-30 09:37:25 AM

vygramul: Where's all the skeptics?


Skeptics of what? The Fahrenheit temperature scale? It's hot-as-balls outside right now.
 
2012-06-30 09:38:37 AM

LouDobbsAwaaaay: vygramul: Where's all the skeptics?

Skeptics of what? The Fahrenheit temperature scale? It's hot-as-balls outside right now.


Global warming.
 
2012-06-30 09:45:46 AM

Gunny Walker: Global warming.


Ugh. I sometimes forget that people frame every discussion about the weather by classifying every event as proof for/against global warming. It's a shame, because there's actually interesting stuff going on right now in a totally-not-related-to-global-warming fashion.

It's like if you wanted to talk about the kinds of food you like or don't like, but all anyone wants to do is classify every ingredient into "fights cancer" or "causes cancer".
 
2012-06-30 09:45:48 AM
The West Coast end to end is normal, the Northwest has been cooler than normal. Nature's air conditioner is blasting full on right now, all my windows are open and its a lovely 70F indoors, 59F outdoors.

People making comments like "the inland West is on fire" seem to have trouble reading maps. One part of Colorado has a bad fire right now. Every year the West has fires. For the Cascade/Sierra ranges, our fire season hasn't even started yet, because it hasn't dried out yet. But it is a normal annual thing.

Whats not normal is the east coast baking in 100F in June, and THAT is why you get the whining about the weather nationally. The East Coast thinks the world revolves around it, and its echo chamber/media noise machine is the loudest in this hemisphere.
 
2012-06-30 09:49:12 AM

Generation_D: People making comments like "the inland West is on fire" seem to have trouble reading maps. One part of Colorado has a bad fire right now. Every year the West has fires.


It's not so much the timing of the fires or the size of them as it is the location. The fires are creeping into some highly-populated areas; I know that some important government supercomputers were shut down because Mesa Labs was too close to the fires (the computers are, unfortunately, very important for weather forecasting).

It's the same with hurricanes; nobody really gives a shiat until they get close to the US coastline.
 
2012-06-30 09:54:08 AM

LouDobbsAwaaaay: Gunny Walker: Global warming.

Ugh. I sometimes forget that people frame every discussion about the weather by classifying every event as proof for/against global warming. It's a shame, because there's actually interesting stuff going on right now in a totally-not-related-to-global-warming fashion.

It's like if you wanted to talk about the kinds of food you like or don't like, but all anyone wants to do is classify every ingredient into "fights cancer" or "causes cancer".


I enjoy every thread about biology being hijacked into a 400 post battle of whether or not God exists.
 
2012-06-30 09:55:48 AM

LouDobbsAwaaaay: Generation_D: People making comments like "the inland West is on fire" seem to have trouble reading maps. One part of Colorado has a bad fire right now. Every year the West has fires.

It's not so much the timing of the fires or the size of them as it is the location. The fires are creeping into some highly-populated areas; I know that some important government supercomputers were shut down because Mesa Labs was too close to the fires (the computers are, unfortunately, very important for weather forecasting).

It's the same with hurricanes; nobody really gives a shiat until they get close to the US coastline.


Its really amazing but if you build near national forests, you will sometimes be at risk for forest fires. Happens every year somewhere in the Cascades or Sierras or foothills in california, some development full of mansions is suddenly a national emergency.
 
2012-06-30 09:59:59 AM

Generation_D: Its really amazing but if you build near national forests, you will sometimes be at risk for forest fires. Happens every year somewhere in the Cascades or Sierras or foothills in california, some development full of mansions is suddenly a national emergency.


Alright, we get it. You don't care about the fires.
 
2012-06-30 10:00:47 AM

LouDobbsAwaaaay: Gunny Walker: Global warming.

Ugh. I sometimes forget that people frame every discussion about the weather by classifying every event as proof for/against global warming. It's a shame, because there's actually interesting stuff going on right now in a totally-not-related-to-global-warming fashion.

It's like if you wanted to talk about the kinds of food you like or don't like, but all anyone wants to do is classify every ingredient into "fights cancer" or "causes cancer".


That's why it was in a tiny font.
 
2012-06-30 10:03:06 AM

LouDobbsAwaaaay: Gunny Walker: Global warming.

Ugh. I sometimes forget that people frame every discussion about the weather by classifying every event as proof for/against global warming. It's a shame, because there's actually interesting stuff going on right now in a totally-not-related-to-global-warming fashion.

It's like if you wanted to talk about the kinds of food you like or don't like, but all anyone wants to do is classify every ingredient into "fights cancer" or "causes cancer".


Boy! We could use some of those falsified global warming numbers right about now!
 
2012-06-30 10:05:06 AM
I'm holed up at work because I don't have power at home and it's 100 degrees outside, so I'm getting a kick out of this thread.
 
2012-06-30 10:08:41 AM

Slaxl: Are Amazon quite bad? I thought these big hosting facilities had umpteen backups and redundancies in place, and this isn't the first time something has happened to knock Amazon using sites off.


Redundancies are nice for things like jet engines and fuel tanks. But most servers worth running contain data and transaction state. The problem with the "cloud" is that yes, the server hardware is distributed. And they can do some pretty cool things with backups, logging data to multiple locations, etc.

But all of those backups have to be transferred over some connection of a finite capacity. If you lose the connection you are still toast. Most redundant storage techniques instantly flip to a failure state that prevents them from taking in new data if it can't talk to all of the mirrors. If you are using distributed storage, yes you can access data locally, but everything from the other parts of the cloud is inaccessible.

And even if you have a complete zero day, turnkey run the system from a remote location arrangement in place, all of the DNS records need to be updates, there are cutover procedures that take a finite amount of time to run, and a legion of host processes that will likely have to be spooled up on the backup system.
 
2012-06-30 10:11:44 AM

LouDobbsAwaaaay: Generation_D: Its really amazing but if you build near national forests, you will sometimes be at risk for forest fires. Happens every year somewhere in the Cascades or Sierras or foothills in california, some development full of mansions is suddenly a national emergency.

Alright, we get it. You don't care about the fires.


Nah, its a big deal if you live in Colorado. Which is 1500 miles from me. I wish them well, our firefighters just sent 30 volunteers, more are on the way.

Our own fire season, which will take place 50-1000 miles from me, won't kick in for another month or so hopefully, or maybe since its been so cool and wet this year the Northwest will not have a bad one at all.

National forests burn, its what they do. The only X factor is the people keep building in places surrounded by it. Want to stop or cut down forest fires, go back to managing the forest like they did for most of the 20th century til environmentalists made them stop: Clear out the undergrowth and cut away the dead. Do it before nature does it for you. We used to do this regularly as part of the Forest Service, but between budget cuts and environmental *concerns* we've stopped.

Also, live in cities where they clear cut the trees a long time ago. Most of us do that, but some can't resist the Call of the Wild. So they drive 20 mi out into the forest and build a house.
 
2012-06-30 10:12:07 AM

Tax Boy: I'm holed up at work because I don't have power at home and it's 100 degrees outside, so I'm getting a kick out of this thread.


My old apartment was serviced by Dominion. And it seemed like three or four once a week we'd get a power sag (enough to drop a game console), and once a year we would lose power for the day.

My new place has NOVEC. And even through that storm, the lights didn't so much as blink.
 
2012-06-30 10:16:58 AM

Generation_D: Also, live in cities where they clear cut the trees a long time ago. Most of us do that, but some can't resist the Call of the Wild. So they drive 20 mi out into the forest and build a house.


I don't think you can accuse Colorado Springs of being a few houses 20 mi out into the forest. The unfortunate placement of some of these fires is especially painful, and it's left tens of thousands evacuating their homes for safety.
 
2012-06-30 10:18:20 AM

Evil Twin Skippy: Tax Boy: I'm holed up at work because I don't have power at home and it's 100 degrees outside, so I'm getting a kick out of this thread.

My old apartment was serviced by Dominion. And it seemed like three or four once a week we'd get a power sag (enough to drop a game console), and once a year we would lose power for the day.

My new place has NOVEC. And even through that storm, the lights didn't so much as blink.


I think it depends a lot more on where you live than who the power company is. In McLean, I'd lose power at least once a month. In Annandale, it rarely happened. Both were Dominion. McLean had a lot fewer below-ground power cables.
 
2012-06-30 10:43:45 AM

IoSaturnalia: This was totally preventable. No datacenter should be at the whims of the grid...ever. Amazon didn't install enough fail-over capablilty in their power distribution or test it properly because that costs money and people are lazy and cheap.


Actually, it sounds like their architecture worked as designed: each of their availability zones is independent from each other, with separate power, data, etc. Failures in one zone won't affect another.

According to their status page all the failures have been localized in one AZ.

Sure, there shouldn't have been power issues in that AZ in the first place -- they should have had more redundancy. Still, shiat happens and even the best-managed datacenters can have issues. That's why Amazon (and common sense) encourages services that require high availability to run services in multiple AZs/regions and set up various failover/replication methods between them. They even provide various methods to assist in doing this.

I seem to recall this happening with Netflix a year or two ago: Amazon had an AZ burp but Netflix wasn't affected because they had services running in multiple AZs and switched instantly to the secondary setup.
 
2012-06-30 10:44:03 AM
Crossing our fingers here in Silver Spring, we still have power, unlike most of Montgomery County. Pepco's website's now down as well. Hopefully our power survives today's storms too.
 
2012-06-30 10:44:32 AM

Generation_D: Whats not normal is the east coast baking in 100F in June, and THAT is why you get the whining about the weather nationally. The East Coast thinks the world revolves around it, and its echo chamber/media noise machine is the loudest in this hemisphere.


It is also not normal to have 118 in Kansas...ever. It is not just the east coast. You seem to think that because you live on the west coast that the only counterpoint to your weather must be the east coast. The entire midwest and much of the southeast, is baking. Where I live has seen freak summers 4 of the last 5, and much of the post-2000 summers have been freakishly dry and well above average temp-wise. You can call it global warming, or regional climate variation or whatever, but our summer weather in Tennessee is markedly different then what it was in the 1980s and 1990s; much of the surrounding area would agree.
 
2012-06-30 10:47:14 AM

LouDobbsAwaaaay: Generation_D: Also, live in cities where they clear cut the trees a long time ago. Most of us do that, but some can't resist the Call of the Wild. So they drive 20 mi out into the forest and build a house.

I don't think you can accuse Colorado Springs of being a few houses 20 mi out into the forest. The unfortunate placement of some of these fires is especially painful, and it's left tens of thousands evacuating their homes for safety.


http://www.denverpost.com/opinion/ci_20954884?source=bb

A U.S. Forest Service analysis found that 40 percent of homes built in the U.S. between 1990 and 2000 were in the WUI. In Colorado, the figure in that time was 50 percent.



looks like it isnt Springs so much as the Wildlife-Urban Interface, which is in the foothills above Springs, and which is a lot of new development.
 
2012-06-30 11:01:26 AM
you know, shiat like this is why i get leery of the companies that want me to put my actual IP data solely on the cloud

fark yea, lets get all my designwork on a remote server where i won't be able to access it if the internet is down here or there
 
Displayed 50 of 84 comments

First | « | 1 | 2 | » | Last | Show all

View Voting Results: Smartest and Funniest


This thread is archived, and closed to new comments.

Continue Farking
Submit a Link »






Report