Company after company I hear about has been pulling their outsourced / South Asian operations back and putting them back in the US. People are tired of doing the needful.

ok how many of you that clicked the link are in IT?

Me

Weaver95: further reading:

As noted in previous Reg stories, it is understood that the error was made when backing out of an upgrade from CA-7 v11.1 to v11.3. The CA-7 upgrade took place at the weekend of 16/17th June and a problem was noticed on Monday which prompted a back-out from the upgrade on Tuesday night. In the back-out, an "inexperienced operator" made the wrong move and the day's data was wiped from the system. This created the backlog that has taken so long to clear.

[singsong voice] I know what they did!

that sounds like after the IPL the operator was bringing the system up and either hit 'cold start' or 'reformat'. they're supposed to reply 'warm start', but it's a common n00b mistake to hit the wrong reply at that point in the IPL. essentially you'll lose about 24hrs of data and have to stop the IPL and reload your backups. Now, assuming you TOOK backups prior to implementing the CA7 upgrade, that's about 5 or 6 hours to a full restore but it sounds like these guys weren't following established procedures and either didn't know what their operator had done OR they borked their backups and had to go back to a previous restore and lost even MORE data. either way, that's extremely sloppy work from their outsourced people. so it probably took them over 12 hours to restore from their mistake, roll CA7 back to previous release and THEN they had to restart the batch cycle and recreate all the data they lost before they could let users back on the system. VERY sloppy and incidentally that probably cost the company a couple/few million bucks.

Might be a drop in prestige, but if you know SQL Server, Access, and Excel VBA like a badass, my company is looking for such people. Most of us work from home.

Came in for a "Please do the needful" reference. Leaving happy.

Did mainframe ops here as well, I think Weaver95 hit the problem spot on.

xanadian: If they're outsourcing jobs, and their outsourcing was responsible for this failure, then I really feel zero sympathy for RBS. They can EABOD.

Yes, but you also have to remember that, in RBS's view, American jobs are outsourcing. (RBS is the Royal Bank of Scotland). So, in reality, it's not the outsourcing per se that's the problem. It's the outsourcing to places where the workers are incompetent at their new jobs, just to save a buck.

Sounds like somebody forgot to do the needful.

Not RBC, but same thing....

RBS has felt what happens when you do the needful, hopefully they learn from it.

/lost two jobs to out sourcing and/or other short-sighted attempts to cut costs
//IT

Your average developer with decent experience in India will cost around $15/hr, which is next to nothing compared to here.

The problem is you need a project management team in both countries to handle the communication. And because of the overhead, projects take longer to complete, and often times require more QA as well. Costs that it seems like many companies don't take into account.

In my experience, it's not worth it to outsource unless it's for the most basic development, and even then it's questionable.

Offshored and even local immigrant tech resources share a set of very consistent flaws, centered around a sense of desperation over keeping the job. Upper management likes this step-and-fetch-it attitude, which they feel contrasts with their "arrogant" and "lazy" American resources. What they don't learn until a year or so out, when most of their projects fail and system reliability suffers, is that desperation is not the proper motivator for delicate and intellectually demanding, self-directed tasks. And, as it turns out, US systems people are not arrogant and lazy, they are honest and thorough.

Offshor-itis symptoms include:

-Lying about project status and hiding issues.
-Being unable to continue work and instead of asking for help, just sitting and doing nothing, hoping the issue will magically resolve itself. Still billing hours, though.
-Approaching every task as if it is a unique problem, creating entirely new solutions where re-using an existing one would suffice. This is because they don't want to have to ask whoever controls the existing solution for help, or risk being noticed somehow.

Their relatively poor education really shows in the consistent application of anti-patterns, rather than meeting to get help deciding how to accomplish something. Offshored codebases really come into their own the first time they have to be refactored or a large feature needs to be added. May as well re-write, since none of the code was designed with their face more than an inch from the monitor, pouring thousands of lines into a problem with a well-known solution to "look busy".

Offshore guys never produce anything your organization can use for anything else, and everything they do produce is brittle and performs poorly. In short, the answer to the executive question about systems: "How hard can it be?" is "actually, very hard indeed!", and offshore resources are just not up to the task.

/sorry Rajesh, but it's true

YixilTesiphon: Might be a drop in prestige, but if you know SQL Server, Access, and Excel VBA like a badass, my company is looking for such people. Most of us work from home.

Also, our job is to gather and transform data from people in a way that is understandable and user-friendly, so coming from the same culture as our clients (i.e. being American) is very, very important.

Weaver95: tdyak: Came in for a "Please do the needful" reference. Leaving happy.

Did mainframe ops here as well, I think Weaver95 hit the problem spot on.

I once saw our mainframe outsource people hit 'reformat' on a regular sunday morning IPL. THAT was pretty interesting...I wasn't sure what the end result would be, since nobody anywhere EVER hits 'reformat' on IPL. took us about two weeks to finally get everything back to working normally.

The worst mistake I ever made was letting the spool file hit 100%. That was a mess that took a couple hours to clean up. The worst/best part was about 8 people were over my shoulder walking ME through fixing it on a step-by-step basis so that (a) I would NEVER do it again, and (b) If I ever saw it happen again I could fix it.

Weaver95: xanadian:
If they're outsourcing jobs, and their outsourcing was responsible for this failure, then I really feel zero sympathy for RBS. They can EABOD.

the real kicker is that based on what I read in that article, this was a completely avoidable situation. outsourcing firms don't always hire well trained individuals, nor do they bother training their people to do the jobs for which they were hired. it is VERY easy to screw up even a routine mainframe IPL...throw in a somewhat more complicated task like backing out of a scheduling upgrade package and combine it with poorly trained people and/or poor communications and control and well...that's just begging for a disaster.

Give the poor guy a break, he was only following the standard Indian Outsourced IT Troubleshooting Script and got hung up on the first question... "Please to tell me, what is your operating system, Windows XP or Windows 7?"

homarjr: and even then it's questionable

Why would any organization introduce such massive risk? It boggles the mind. In the US, you have direct and local access to the best IT people in the world. We literally invented IT and have dominated it for decades. Instead of counting your lucky stars and scooping up the world's best talent, you spit in the face of US IT pros and "give it a whirl" with third-rate knock-off IT people that work bizarre hours and can barely be understood.

It is the essence of short-sightedness. "Why not? What's the worst that can happen?"

Weaver95: I once saw our mainframe outsource people hit 'reformat' on a regular sunday morning IPL. THAT was pretty interesting...I wasn't sure what the end result would be, since nobody anywhere EVER hits 'reformat' on IPL. took us about two weeks to finally get everything back to working normally.

So why isn't there an "Are you sure?" dialog on that option?

Maybe one that requires you to solve a captcha even.

/Especially if the two words are something like "Losta Wholeweekswork"

YixilTesiphon: YixilTesiphon: Might be a drop in prestige, but if you know SQL Server, Access, and Excel VBA like a badass, my company is looking for such people. Most of us work from home.

Weaver95: incidentally, if anyone is looking to hire an IT systems analyst with over 15 years experience drop me a line and i'll forward a resume to your corporate/company website and/or HR department.

This is Fark, not reverse craigslist.

My favorite part FTA:
Though there were competent people working there, our sources said quality of work from India was patchy and that they had raised these problems with RBS management for the past two years.

Management knew there were issues for years, they had it coming. Nothing to see here, move along.

mccallcl: homarjr: and even then it's questionable

Why would any organization introduce such massive risk? It boggles the mind. In the US, you have direct and local access to the best IT people in the world. We literally invented IT and have dominated it for decades. Instead of counting your lucky stars and scooping up the world's best talent, you spit in the face of US IT pros and "give it a whirl" with third-rate knock-off IT people that work bizarre hours and can barely be understood.

It is the essence of short-sightedness. "Why not? What's the worst that can happen?"

Salary is an operational expense. Director-level and up gets rewarded by managing down operational expenses. Director level and above rarely understands the nuts and bolts of what it is their people do. They are busy looking at heat maps and trend lines. Thus, outsourcing looks great on paper. And in a year or two later, if outsourcing was a mistake, they can always sell the company which by now has great bottom line operational expenses.

It can take a while for terrible outsourced performance to manifest itself. Businesses are very short sighted, managing quarter-to-quarter usually. So the Directors look like heroes and the company shows great numbers -- til it all comes crashing down in a few quarters later if there's a mass customer defection.

By then you're supposed to have sold the company and pocketed the loot anyway.

Weaver95: A lot of companies want efficiently run IT operations

Eeehhh. I would say a lot of companies want "functional" IT operations. It seems like even now, twenty years deep into the digital workplace, few people truly understand the vast complexity of IT operations and even fewer people understand how to organize that complexity into an efficient and reliable stream of processes. As long as email works, nobody cares how or why, and it certainly doesn't have to be efficient.

I don't know what it's like in shops that are dominated by mainframes like you're describing, but in the standard business world where the IT department's function is basically "make sure email and internet work all the time", it's still awful sloppy. I can't even convince half the people I work with to stop running application servers like Exchange or SQL with domain administrator credentials. If it ever came down to something like this it wouldn't be "gather the team and implement a plan", it would be "hey, Vegan, we seriously farked up the entire system and now we're taking off for the weekend, have it fixed on your own with no help by Monday morning when we get back or you're fired.... oh, and don't change my service accounts to unprivileged accounts while we're away, I don't even know how my own stuff works so we can't take even basic steps to ensure this doesn't happen again".

I guess my point here is that IT is still widely b0rked even when you're not cutting your lifers who practically built the thing and sending their complex work to untrained south-Asian recent grads. It's okay to blame outsourcing for things like this, but I think a lot of tech people have swollen heads and ignore that, realistically, a lot of IT on our own shores is just as defective to begin with.

Weaver95: Vlad_the_Inaner: Weaver95: I once saw our mainframe outsource people hit 'reformat' on a regular sunday morning IPL. THAT was pretty interesting...I wasn't sure what the end result would be, since nobody anywhere EVER hits 'reformat' on IPL. took us about two weeks to finally get everything back to working normally.

So why isn't there an "Are you sure?" dialog on that option?

Maybe one that requires you to solve a captcha even.

/Especially if the two words are something like "Losta Wholeweekswork"

we had built in a series of 'are you sure' replies into the IPL decision points but...once we handed things over to the outsource people, if someone was determined to screw things up then all they had to do was just sit there on the main console and give the wrong replies at the right time.

So you basically gave them a "Choose your own adventure" book?

I had to type in the commands based off of a 3" thick handwritten notebook that covered all possible problems and outcomes. With scribbled notes in the margins you needed a magnifying glass to read.

The only thing that was nicely laid out was the HMC, and that was on OS/2.

mccallcl: homarjr: and even then it's questionable

Why would any organization introduce such massive risk? It boggles the mind. In the US, you have direct and local access to the best IT people in the world. We literally invented IT and have dominated it for decades. Instead of counting your lucky stars and scooping up the world's best talent, you spit in the face of US IT pros and "give it a whirl" with third-rate knock-off IT people that work bizarre hours and can barely be understood.

It is the essence of short-sightedness. "Why not? What's the worst that can happen?"

It's because developers are just monkeys, so to speak. Management often has this idea that they know exactly what they want, and when they get just "anyone" to do it for them, it can be a disaster. IT teams need a back-and-forth type of communication because the IT team knows way more about how a system operates and what it's capable of than management does, in nearly every single case.

It's the same thing with sales. The salesforce knows the product inside and out. They know why people like it and probably how it can be improved. This shouldn't be news to anyone.

Bad management is just people who think they know everything. Good management is people who surround themselves with people who collectively know everything, and use all the information they can get to move forward.

I learned this in my first year of business school. It seems like such common knowledge, but it somehow isn't. It's ridiculous.

Vegan Meat Popsicle: Weaver95: A lot of companies want efficiently run IT operations

Eeehhh. I would say a lot of companies want "functional" IT operations. It seems like even now, twenty years deep into the digital workplace, few people truly understand the vast complexity of IT operations and even fewer people understand how to organize that complexity into an efficient and reliable stream of processes. As long as email works, nobody cares how or why, and it certainly doesn't have to be efficient.

I don't know what it's like in shops that are dominated by mainframes like you're describing, but in the standard business world where the IT department's function is basically "make sure email and internet work all the time", it's still awful sloppy. I can't even convince half the people I work with to stop running application servers like Exchange or SQL with domain administrator credentials. If it ever came down to something like this it wouldn't be "gather the team and implement a plan", it would be "hey, Vegan, we seriously farked up the entire system and now we're taking off for the weekend, have it fixed on your own with no help by Monday morning when we get back or you're fired.... oh, and don't change my service accounts to unprivileged accounts while we're away, I don't even know how my own stuff works so we can't take even basic steps to ensure this doesn't happen again".

I guess my point here is that IT is still widely b0rked even when you're not cutting your lifers who practically built the thing and sending their complex work to untrained south-Asian recent grads. It's okay to blame outsourcing for things like this, but I think a lot of tech people have swollen heads and ignore that, realistically, a lot of IT on our own shores is just as defective to begin with.

We're moving to the Cloud next quarter, I don't know why you spend time upgrading those legacy systems.

Sir Not Sure The Unscannable: This is Fark, not reverse craigslist.

Then GTFO if you don't like it, punk. Those two have been here long before you showed your ass around here, and contribute more to a single thread than you ever will.

/no one talks down to the Weave.

Login: Sir Not Sure The Unscannable
Fark account number: 786379
Account created: 2012-05-19 01:58:54

noob.

Hawnkee: [verydemotivational.files.wordpress.com image 480x366]

shouldn't they be single file?

/queue blasters and weird sandpeople shreek

You don't see this often in the US. As in no one reports that your problem was caused by outsourcing. Must be against the law to say that somewhere.
Anyway, that is what makes this article interesting, that someone wrote about why it happened.