Do you have adblock enabled?
 
If you can read this, either the style sheet didn't load or you have an older browser that doesn't support style sheets. Try clearing your browser cache and refreshing the page.

(The Next Web)   University server accidentally reformats all computers campus-wide, including itself. Little Bobby Tables strikes again   (thenextweb.com ) divider line
    More: Scary, Emory University, Windows, servers  
•       •       •

6637 clicks; posted to Geek » on 17 May 2014 at 2:43 PM (1 year ago)   |   Favorite    |   share:  Share on Twitter share via Email Share on Facebook   more»



100 Comments   (+0 »)
   
View Voting Results: Smartest and Funniest

Archived thread

First | « | 1 | 2 | » | Last | Show all
 
2014-05-17 12:11:34 PM  
imgs.xkcd.com
 
2014-05-17 12:22:16 PM  
How in the hell does this even happen?  Wouldn't your servers be segmented off from the workstations in AD?
 
2014-05-17 12:31:59 PM  

Lsherm: How in the hell does this even happen?  Wouldn't your servers be segmented off from the workstations in AD?


Among other things, and who sends out an unattended upgrade command to any workstation?
 
2014-05-17 12:37:10 PM  
Two more things:

1)  Now is when they discover any holes in their DR plan.

2)  I wonder how many IT job openings they have.
 
2014-05-17 12:41:54 PM  
Hope any and all researchers use something like Google Drive to back up their shiat.
 
2014-05-17 12:42:34 PM  
Well.  That makes me feel better about the mistakes I make.

The most painful one wasn't really my fault.  Lost 2 RAID 5 drives at the same time.  In a virtual host.  That's when backups come in handy.  But that's not important.
 
2014-05-17 12:52:30 PM  

I_Am_Weasel: Well.  That makes me feel better about the mistakes I make.

The most painful one wasn't really my fault.  Lost 2 RAID 5 drives at the same time.  In a virtual host.  That's when backups come in handy.  But that's not important.


A certain vintage of Compaq servers were known for marking multiple RAID disks as bad at the same time. The amount of overtime I made that year before Compaq found the culprit - a missing capacitor on the drive bay backplate - and replaced 60 servers at their own cost, was enough to allow me to make a downpayment on a house.

Worst "Oops" I ever made was typing "clear arp" in a Cisco 7500 router in the middle of the day, causing it to crash ad reboot.
 
2014-05-17 12:59:29 PM  
Computers don't accidently anything.  People accidently stuff.
 
2014-05-17 01:04:58 PM  
Lemme guess.  Misconfig in WDS.
 
2014-05-17 01:06:18 PM  
And they didn't have backups?  Someone's getting a firing...

/which reminds me...I have to pester the Powers That Be to cough up the dough for our backup regime...
 
2014-05-17 01:09:07 PM  

I_Am_Weasel: Well.  That makes me feel better about the mistakes I make.

The most painful one wasn't really my fault.  Lost 2 RAID 5 drives at the same time.  In a virtual host.  That's when backups come in handy.  But that's not important.


I had something similar happen:  2 disks with a "puncture" error, one right after the other.  Apparently, RAID will also spread errors across multiple drives, not just data or algorithms.  At least, in this case.  Backups saved the day there.  But, I missed a step in the AD restore process, so farked up the instance of AD on that machine.  No data loss, just some extra work.

/also helps to have more than 1 domain server
 
2014-05-17 02:46:44 PM  
That happened to me.  IT sent out a Windows 7 upgrade command, it tried to upgrade my Windows XP box by reformatting first.  Fortunately I do the majority of my work in Linux.  It gained me a week of peace and quiet because I couldn't access the company email (Exchange).
 
2014-05-17 02:53:54 PM  
The employee's name was George P. Burdell, right?
 
2014-05-17 03:01:25 PM  
dilbert.com
 
2014-05-17 03:13:48 PM  
H-1B visa quality work right there.
 
2014-05-17 03:22:40 PM  

Ambivalence: Computers don't accidently anything.  People accidently stuff.


The whole thing?
 
2014-05-17 03:26:11 PM  
Microsoft product..... No room for mistakes as they are built in...
 
2014-05-17 03:29:39 PM  
"Have you ever reformatted a computer and then immediately realized you shouldn't have?"


...nope, can't say that I have ever done that.
 
2014-05-17 03:32:50 PM  

Pharque-it: Microsoft product..... No room for mistakes as they are built in...


SCCM is pronounced SCUMMY for a reason.  Of course, we use Altiris which is an offshoot of it and isn't much better it would seem.

/not really
//it's all how it's set up and who touches it
 
2014-05-17 03:33:03 PM  

BizarreMan: Lsherm: How in the hell does this even happen?  Wouldn't your servers be segmented off from the workstations in AD?

Among other things, and who sends out an unattended upgrade command to any workstation?


Which large universities enable university-wide image deployment from central IT?

I am employed as a computer administrator for a department within a medium-sized university. IT manages an SCCM system, but control over image deployment is delegated to the support staff within each university department, not to the central IT management.

/I actually block deployment of SCCM within my area because I am whacked-out paranoid anyway, so even were something like this to happen where I work my systems would remain completely unaffected.
 
2014-05-17 03:41:18 PM  

RTOGUY: "Have you ever reformatted a computer and then immediately realized you shouldn't have?"


...nope, can't say that I have ever done that.


We used to say there were two types of users. Those who had formatted their hard drive, and those who were going to. . .
 
2014-05-17 03:51:28 PM  
I'd bet it was some grad student working for the IT department.
 
2014-05-17 03:51:30 PM  

RTOGUY: "Have you ever reformatted a computer and then immediately realized you shouldn't have?"


...nope, can't say that I have ever done that.


I've reformatted a computer and someone else realized I shouldn't have.

People, when I tell you, "tell me what data you want to save," it's because the things you don't tell me about are going to disappear forever.
 
2014-05-17 03:56:58 PM  
Couldn't back up the contents of an personally "encrypted" folder on a VP's c: drive... had great back ups all over the company, could do real nice restores, etc... but damm, when his hard drive failed; I restored everything... and an empty folder right there at c:\"fire the IT guy if this fails"... oops.
 
2014-05-17 03:57:06 PM  

yukichigai: RTOGUY: "Have you ever reformatted a computer and then immediately realized you shouldn't have?"


...nope, can't say that I have ever done that.

I've reformatted a computer and someone else realized I shouldn't have.

People, when I tell you, "tell me what data you want to save," it's because the things you don't tell me about are going to disappear forever.


'What? You don't have a copy of my hard drive? I thought you did that! I saw a commercial/movie/show that did that!"
 
2014-05-17 03:59:03 PM  
Can't laugh hard enough for this. Good going, emory.
 
2014-05-17 03:59:29 PM  
Well, did they think of unplugging their computers then plugging them back in?

/buncha ivory tower eggheads, honestly...
 
2014-05-17 04:05:51 PM  

RTOGUY: "Have you ever reformatted a computer and then immediately realized you shouldn't have?"


...nope, can't say that I have ever done that.


A long time ago, I spent four hours onsite at what was then Rolls-Royce Nuclear (in Derby, IIRC) installing and configuring an HP1000 system ready for a training class the next day. It was close to midnight when I was done, and all that remained was to backup the disk, just in case. Those particular systems had a wonderful feature: a single-button backup that allowed you to dump a disk image to a tape cartridge.

Or, if you pressed the other button right next to it, allowed you to dump a blank tape cartridge to your carefully prepared disk image.

The remainder of this story is left as an exercise for the reader.
 
2014-05-17 04:14:33 PM  

Ambivalence: Computers don't accidently anything.  People accidently stuff.


They can, sometimes, but it generally requires faulty hardware.
 
2014-05-17 04:15:52 PM  
Note to self:

No hiring of any IT people with "Emory University" on their resume.
 
2014-05-17 04:19:47 PM  
A couple years ago i visited a customer running QC instrumentation in a regulated/validated environment, who religiously backed up the C:\ partition of his HDD to the D:\ partition of his HDD... You know, incase the drive ever failed. Guy turned whiter than paper when i told him those were partitions on the same physical drive.
 
2014-05-17 04:33:47 PM  

fluffy2097: H-1B visa quality work right there.


Ha!
Good one.

Made me spill my drink.
Thank goodness not on the keyboard.
 
2014-05-17 04:41:02 PM  

AngryDragon: Note to self:

No hiring of any IT people with "Emory University" on their resume.


Note to self:

No more Windows/Microsoft Servers installed.
 
2014-05-17 04:50:14 PM  
I wonder if someone with access to the system images was recently fired, and if they weren't walked out of the building the moment they were informed?
 
2014-05-17 05:15:22 PM  
FTFA: UPDATE: 12 PM, May 15, 2014
At this time, we have a more automated deployment that has been tested on multiple devices. Some models of laptops do not quite work, but we are cleaning those up as they are discovered. This deployment will still require a visit from a technician but should be much quicker to complete.
This development process has taken longer than expected and while we still hope to be completed with all machine recoveries today, some machines may need to be completed tomorrow morning.
UPDATE: 10 AM, May 15, 2014
We are developing a more automated approach and technicians are testing it now.


Good to see that they've learned from this experience.  Clearly, they need a more automated image deployment system.
 
2014-05-17 05:23:46 PM  
NEVER do auto updates like that! OMG..That is just asking for that kind of disaster. You may think it's real cool to just sit in your office and push a button and refresh machines, but that power
cuts both ways when something like this happens. If you force an update/image/etc.. to some machine a professor, who didn't have a backup of their presentation/research data, was using..It doesn't matter that they are stupid for doing that..YOU are the one that is going to pay when their department head or dean starts screaming.. Just because you can, doesn't mean you should.
 
2014-05-17 05:26:26 PM  

czetie: A long time ago,

It's stories like this that keep me around here.
 
2014-05-17 05:28:06 PM  

Dimensio: BizarreMan: Lsherm: How in the hell does this even happen?  Wouldn't your servers be segmented off from the workstations in AD?

Among other things, and who sends out an unattended upgrade command to any workstation?

Which large universities enable university-wide image deployment from central IT?

I am employed as a computer administrator for a department within a medium-sized university. IT manages an SCCM system, but control over image deployment is delegated to the support staff within each university department, not to the central IT management.

/I actually block deployment of SCCM within my area because I am whacked-out paranoid anyway, so even were something like this to happen where I work my systems would remain completely unaffected.


Depends on the University - I own the desktops across the entire campus of a small/medium public University. My staff support the entire campus at the desktop level, and I could technically reproduce the incident in the article with my LDMS system if I tried hard enough.

Not everyone runs decentralized IT. We're actively trying to recentralize all IT purchasing at the moment in fact, because the "departments make their own choices" version turned into such an ego driven, tax dollar wasting clusterfark.
 
2014-05-17 05:31:19 PM  

Pharque-it: AngryDragon: Note to self:

No hiring of any IT people with "Emory University" on their resume.

Note to self:

No more Windows/Microsoft Servers installed.


Because clearly it wasn't the fault of the IT staff.
 
2014-05-17 05:41:28 PM  
I have just now looked over at my backup hard drive with a big smile and then a big HA HA to that universities IT staff.
 
2014-05-17 05:42:00 PM  

wildcardjack: I'd bet it was they manage to pin it on some grad student working for the IT department.

 
2014-05-17 05:43:53 PM  

Pharque-it: AngryDragon: Note to self:

No hiring of any IT people with "Emory University" on their resume.

Note to self:

No more Windows/Microsoft Servers installed.


Why? How was this the fault of either Microsoft or Windows server?
 
2014-05-17 05:49:56 PM  
What does this button do?
 
2014-05-17 05:53:47 PM  

SpdrJay: What does this button do?


Make Babby
 
2014-05-17 05:56:10 PM  

SpdrJay: What does this button do?


That's button that makes a new backup of the network storage drives, over-writing the old backup from two weeks ago with a new backup of completely empty reformatted drives.

Well done.
 
2014-05-17 05:59:32 PM  

BizarreMan: 2) I wonder how many IT job openings they have.


My thoughts as well. We can expect the firing/stepping down of their CTO/ CIO (or equivalent), as well as a few other higher ups, and maybe a couple low level guys who happened to be in the same room when it happened (just to make it look "thorough").
 
2014-05-17 06:03:33 PM  

czetie: Or, if you pressed the other button right next to it, allowed you to dump a blank tape cartridge to your carefully prepared disk image.

The remainder of this story is left as an exercise for the reader.


He fixes the cable?
 
2014-05-17 06:03:49 PM  

RoxtarRyan: BizarreMan: 2) I wonder how many IT job openings they have.

My thoughts as well. We can expect the firing/stepping down of their CTO/ CIO (or equivalent), as well as a few other higher ups, and maybe a couple low level guys who happened to be in the same room when it happened (just to make it look "thorough").


I work for a university.  It's very difficult to fire someone and since they had backups I doubt any heads will roll.  Someone might get written up and I'm sure there's going to be an army of consultants brought in to make sure it can never happen again.
 
2014-05-17 06:07:33 PM  
Destruct sequence 1, code 1, 1A
Destruct sequence 2, code 1, 1A, 2B
Destruct sequence 3, code 1B, 2B, 3
Code 000, destruct 0

movies.trekcore.com
 
2014-05-17 06:08:28 PM  

Mr. Shabooboo: NEVER do auto updates like that! OMG..That is just asking for that kind of disaster. You may think it's real cool to just sit in your office and push a button and refresh machines, but that power
cuts both ways when something like this happens. If you force an update/image/etc.. to some machine a professor, who didn't have a backup of their presentation/research data, was using..It doesn't matter that they are stupid for doing that..YOU are the one that is going to pay when their department head or dean starts screaming.. Just because you can, doesn't mean you should.


My method of system imaging involves deploying a sysprepped image onto a spare hard drive and swapping the spare drive with the active drive in the system to be imaged.

I have considered implementing a more automated imaging system that would allow deployment of images over a network, but I have considered several practical reasons not to do so:

1) Systems are typically only reimaged when staff is replaced (which usually requires physically moving the computer anyway, meaning that physical access to the computer is already occurring) or when hardware fails (which, again, means that physical access is already occurring).

2) I insist upon retaining a backup of the previous system configuration for a set period of time. While I could automate backing up a system and then imaging, a hard drive swap allows the system to be available for use much more quickly, and I am able to back up the removed hard drive at my leisure.

Were I managing more than 100 workstations, a network imaging solution would probably be preferable, but at present I am able to manage system deployment through drive swaps.
 
Displayed 50 of 100 comments

First | « | 1 | 2 | » | Last | Show all

View Voting Results: Smartest and Funniest


This thread is archived, and closed to new comments.

Continue Farking
Submit a Link »
On Twitter






In Other Media


  1. Links are submitted by members of the Fark community.

  2. When community members submit a link, they also write a custom headline for the story.

  3. Other Farkers comment on the links. This is the number of comments. Click here to read them.

  4. Click here to submit a link.

Report