A week, a whole week. Imagine if this were a military contract...

Shoulda called Geek Squad.

At this point, I think Geek Squad could have done a better job. This contract with Northrop Grumann has been an ongoing clusterfark.

That's pretty pathetic. Don't they have a DR plan? WTF?

Almost every company I've worked for that outsourced ends up regretting it.

It's a rock and a hard place. Government worker or contractor? Either way, your boned.

The private sector is always more efficient.

Wash
Rinse
Repeat

What "Unusable DMV computers" may look like

Northrup Grumman is basically a government agency.

They are just as slow an inept as anything .gov

slayer199: Don't they have a DR plan?

Probalby not for 485 servers out of 4800. I'm sure the mainframes and a dozen or so servers are covered by a DR plan. And I'm also pretty sure that the DR plan in question forgets a pile of very essential things that make it unusable in a real life scenario.

[CSB time]
One of my customers' DR plan relied on reprogramming the PBX to forward all calls to the DR location. During the 2003 power outage that affected Ontario and the NE US, they quickly found out that this only works if the PBX still has power. It also helps, when you decide to outsource your call center management, to ensure that the company you outsource to has generators!
[/CSB]

I don't know if this is particularly indicative of the level of computer service you get when you outsource to an airplane manufacturer, but I've seen the same level of incompetence from Lockheed Martin's IT consultants.

Flab: slayer199: Don't they have a DR plan?

Probalby not for 485 servers out of 4800. I'm sure the mainframes and a dozen or so servers are covered by a DR plan. And I'm also pretty sure that the DR plan in question forgets a pile of very essential things that make it unusable in a real life scenario.

[CSB time]
One of my customers' DR plan relied on reprogramming the PBX to forward all calls to the DR location. During the 2003 power outage that affected Ontario and the NE US, they quickly found out that this only works if the PBX still has power. It also helps, when you decide to outsource your call center management, to ensure that the company you outsource to has generators!
[/CSB]

The article mentioned a SAN failure.

For that many servers to be fubar at the same time it would have to be a SAN fabric failure. You're typically not going to connect 485 servers to ONE SAN... unless it is one helluva SAN. If it is one bigass SAN, it would have that redundancy built-in.

Since this is NG we're talking about, they probably connected to 485 servers to an iSCSI SAN built on top of FreeNAS, using Extreme Networks with Spanning-tree instead of EAPS rings as the backbone.

How did I come to that conclusion? Simple... I thought of the worst way possible you could connect 485 servers to a storage platform, and determined that was the "Northrup Grumman Preferred Architecture".

I blame Bush. LAUGHTER OL. Hello this was the joke you see I was saying I blame Bush which is what people say when they whine about the politics and the economy not being better today but I use this in the context of the story because the CEO Officer of this company is named Mister Wesley W.W. Bush.

timujin: I don't know if this is particularly indicative of the level of computer service you get when you outsource to an airplane manufacturer, but I've seen the same level of incompetence from Lockheed Martin's IT consultants.

I've seen that same incompetence in most outsourcing companies.

For some reason companies think they can outsource, save money, and still run an effective IT group. The upside for outsourcing is that it is cheap. The downside is that any problem is now magnified.

An outsource company still has to provide you the same level of service you provided before, but has to do it for half the cost in order to make it cheaper for you. In order to cut the cost in half, you get sh*t designs deployed by sh*t engineers managed by sh*t project managers, that is supported by sh*t tech-support in a sh*t country.

meow said the dog: I blame Bush. LAUGHTER OL. Hello this was the joke you see I was saying I blame Bush which is what people say when they whine about the politics and the economy not being better today but I use this in the context of the story because the CEO Officer of this company is named Mister Wesley W.W. Bush.

You really should only shoot heroin that has been stepped on... this pure stuff you're doing is frying your brain.

Cubansaltyballs: meow said the dog: I blame Bush. LAUGHTER OL. Hello this was the joke you see I was saying I blame Bush which is what people say when they whine about the politics and the economy not being better today but I use this in the context of the story because the CEO Officer of this company is named Mister Wesley W.W. Bush.

You really should only shoot heroin that has been stepped on... this pure stuff you're doing is frying your brain.

There is an emerging technique in her curious stylings.

2wolves: There is an emerging technique in her curious stylings.

I saw that. It was lame... only chasing the dragon could generate such inanity.

Cubansaltyballs: Since this is NG we're talking about, they probably connected to 485 servers to an iSCSI SAN built on top of FreeNAS, using Extreme Networks with Spanning-tree instead of EAPS rings as the backbone.

It was an EMC SAN. Not sure which one, but EMC is saying that whatever happened to their SAN hasn't ever happened before.

I'm not quite buying that.

jbuist: Cubansaltyballs: Since this is NG we're talking about, they probably connected to 485 servers to an iSCSI SAN built on top of FreeNAS, using Extreme Networks with Spanning-tree instead of EAPS rings as the backbone.

It was an EMC SAN. Not sure which one, but EMC is saying that whatever happened to their SAN hasn't ever happened before.

I'm not quite buying that.

I'd buy that. This is probably the first time they've had a bigass SAN get managed by someone that incompetent. So technically, it may have never happened before.

When you're dealing with incompetence at these staggering levels, "Unique" is a politically correct buzzword for "f*cking retarded"

[cool story time]
I did a project last week where I put in a new core network. Their old network used two 6509s with SUP720-10G as a 4-port router. One L3 interface would have 10 switches daisy-chained off it and called a VLAN. That is how they did inter-VLAN routing. When they wanted me to setup their 4x Nexus 7000s(approx $1.5Mil total) in the same way... my response was "The Nexus switches are better suited with a design that isn't so unique". In my head I though "If I ever come across a Leprechaun, I'm asking for a pardon... because you are too dumb to live and I have to murder you for the sake of humanity"

[end cool story time, Bro]

Kirk's_Toupee: I wonder if the idiots at NG thought that they didn't need to back-up things because they have raid servers. (i have no idea if they do)

Yes they do have these.

Wait, wait, I thought the private sector was better and more efficient at anything the government does. It's like Republicans have been lying about that the whole time.

Flab: Probalby not for 485 servers out of 4800. I'm sure the mainframes and a dozen or so servers are covered by a DR plan. And I'm also pretty sure that the DR plan in question forgets a pile of very essential things that make it unusable in a real life scenario.

I have to admit, when I think of Northrop-Grumman I don't think of an IT Services Provider. I think of jets. Still, a good IT policy is to eliminate single-points of failure...and I'm just as blown away by the SAN outage as I am that it lasted AN ENTIRE WEEK? How the hell do you have a SAN failure that lasts an entire week?

All of our production environment data is replicated to our DR site which is 500 miles away and VMs are protected with SRM (that was a FUN project to run...NOT). You mean to tell me nobody at NG thought of these basic things? And wouldn't Virginia have stiff penalties written into the contract for any prolonged outages? If they didn't, shame on them.

Cubansaltyballs: Since this is NG we're talking about, they probably connected to 485 servers to an iSCSI SAN built on top of FreeNAS, using Extreme Networks with Spanning-tree instead of EAPS rings as the backbone.

ROFL. That's really a funny yet horrible at the same time.

GAT_00: Wait, wait, I thought the private sector was better and more efficient at anything the government does. It's like Republicans have been lying about that the whole time.

Where does NG get most of their business?

Cubansaltyballs: The article mentioned a SAN failure.

For that many servers to be fubar at the same time it would have to be a SAN fabric failure.

Maybe the SAN that failed knocked off a few key servers (PDCs, DNSes, etc..) and the problem propagated to the other ones.

But yeah, their SAN design really sucks if it knocked off 500 servers.

slayer199: Where does NG get most of their business?

So we're now calling private companies government run because they get government contracts?

GAT_00: slayer199: Where does NG get most of their business?

So we're now calling private companies government run because they get government contracts?

No, we are calling them government run because they are so dependent on government that they are unable to compete in the open market.

slayer199: How the hell do you have a SAN failure that lasts an entire week?

I'm guessing maddly swapping parts ond doing microcode upgrades, hampered by 15 levels of NG management interacting with VA's govt representatives. Hourly "touch-base" status calls where techies have to go over what has happened so far and start over with "Computer terminology for dummies" because three more people from Governance & Contract Managment have joined and are asking silly questions.

Total duration: 2 days.

Server and application people slowly restarting their systems and in many cases, restoring databases from earlier "known good" saves.

Total duration: 5 days or more, depending on how hard was the SAN crash.

Followed by: What do you mean we don't have a backup? That Oracle "live backup" agent that we didn't buy was important?

Crosshair: No, we are calling them government run because they are so dependent on government that they are unable to compete in the open market.

Because there's such a big private market for fighter planes?

They are a private company. Quit dancing around because you can't stand the idea of a private company screwing up a government outsourced project.

Flab: Cubansaltyballs: The article mentioned a SAN failure.

For that many servers to be fubar at the same time it would have to be a SAN fabric failure.

Maybe the SAN that failed knocked off a few key servers (PDCs, DNSes, etc..) and the problem propagated to the other ones.

But yeah, their SAN design really sucks if it knocked off 500 servers.

Unless the SAN is in a QA Lab or Dev Lab, it should be fully redundant. Redundant power supplies, processors, SAN fabrics, etc..

Everything should be survivable.

Flab: slayer199: How the hell do you have a SAN failure that lasts an entire week?

I'm guessing maddly swapping parts ond doing microcode upgrades, hampered by 15 levels of NG management interacting with VA's govt representatives. Hourly "touch-base" status calls where techies have to go over what has happened so far and start over with "Computer terminology for dummies" because three more people from Governance & Contract Managment have joined and are asking silly questions.

Total duration: 2 days.

Server and application people slowly restarting their systems and in many cases, restoring databases from earlier "known good" saves.

Total duration: 5 days or more, depending on how hard was the SAN crash.

Followed by: What do you mean we don't have a backup? That Oracle "live backup" agent that we didn't buy was important?

I've seen this happen to hundreds of servers because the DC was flooded or burnt down.

The only way this large of a failure could happen outside of a disaster, is mass incompetence.

I think NG's mission statement includes their dedication to incompetence.

GAT_00: So we're now calling private companies government run because they get government contracts?

I'm essentially saying that they're an extension of the government as the vast majority of their business comes from the government. Do you think this may have something to do with it?

Seriously...when you think of IT Services, do you think of NG?

slayer199: GAT_00: So we're now calling private companies government run because they get government contracts?

I'm essentially saying that they're an extension of the government as the vast majority of their business comes from the government. Do you think this may have something to do with it?

Seriously...when you think of IT Services, do you think of NG?

They are a private company. Stop trying to dodge the obvious because you can't stand the reality.

The government run servers were better than the private outsourced servers.

Cubansaltyballs: Unless the SAN is in a QA Lab or Dev Lab, it should be fully redundant. Redundant power supplies, processors, SAN fabrics, etc..

Up to a few years ago, IBM Shark SANs were redundant-everything... except the one relay that flips between the primary power supply and the secondary power supply. The engineers has decided that the MTBF of that part was higher than the risk of having two relays flip in opposite ways and end up with 0 power-supplies feeding the Shark.

Guess which part broke?

The only way this large of a failure could happen outside of a disaster, is mass incompetence.

I'm in full agreement with you.

GAT_00: They are a private company. Stop trying to dodge the obvious because you can't stand the reality.

The government run servers were better than the private outsourced servers.

Yes. That is a product of outsourcing.

When someone outsources, you multiply the worst of the two things and that is your result.

When Govt outsources to private industry, you have tons of red tape mixed with extra bureaucracy multiplied by the cheapest way possible.

Since NG has basically been a privately run arm of the govt for 60 years, you run that same formula, but you square the result so you have regular outsourcing headaches ^2.

Cubansaltyballs: GAT_00: They are a private company. Stop trying to dodge the obvious because you can't stand the reality.

The government run servers were better than the private outsourced servers.

Yes. That is a product of outsourcing.

When someone outsources, you multiply the worst of the two things and that is your result.

When Govt outsources to private industry, you have tons of red tape mixed with extra bureaucracy multiplied by the cheapest way possible.

Since NG has basically been a privately run arm of the govt for 60 years, you run that same formula, but you square the result so you have regular outsourcing headaches ^2.

I like how it's all the government's fault a private company's servers failed.

GAT_00: I like how it's all the government's fault a private company's servers failed.

It's not. It's the culture of failure that permeates NG.

That culture of fail is cultivated by the govt, but multiplied by a private company's fail. in this case it is fail^2.

Cubansaltyballs: GAT_00: I like how it's all the government's fault a private company's servers failed.

It's not. It's the culture of failure that permeates NG.

That culture of fail is cultivated by the govt, but multiplied by a private company's fail. in this case it is fail^2.

So the government is to blame? You're agreeing with me. Thanks.

kingoomieiii: What "Unusable DMV computers" may look like

All of my experiences with the DMV - that looks like that status quo.

GAT_00: So the government is to blame? You're agreeing with me. Thanks.

No. NG is to blame.

Cubansaltyballs: GAT_00: So the government is to blame? You're agreeing with me. Thanks.

No. NG is to blame.

So the private company can't do the job as well as the government can. Still agreeing with me.

Anybody have any idea who came up with the idea contract with NG for this?

I'll be here listening to the crickets.....

GAT_00: Cubansaltyballs: GAT_00: So the government is to blame? You're agreeing with me. Thanks.

No. NG is to blame.

So the private company can't do the job as well as the government can. Still agreeing with me.

No. Companies and govt f*ck up. It is part of the equation. Govt f*cks up as private companies.

NG is a hybrid, so the f*ck up is ^2.