Skip to content
Do you have adblock enabled?
 
If you can read this, either the style sheet didn't load or you have an older browser that doesn't support style sheets. Try clearing your browser cache and refreshing the page.
One last bit of database hardware maintenance tonight
Posted by Mike at 2007-12-03 5:29:06 PM, edited 2007-12-03 5:31:03 PM (23 comments) | Permalink
More: FarkBlog

•       •       •

2 clicks; posted to Main » on 03 Dec 2007 at 5:29 PM (15 years ago)   |   Favorite    |   share:  Share on Twitter share via Email Share on Facebook



There will be one last round of database downtime tonight at 9 pm Eastern Time. This time we'll be replacing the temporary motherboard we put in Friday with a permanent replacement.

It'll take about 30 minutes -- or probably less. Anyone who's built PC's knows how that goes. :) I've already got the CPU, fan, and RAM installed on the board already, so it's just a matter of pulling the machine out of the rack, swapping boards, hooking SCSI/PATA/USB/power LED/power switch/reset switch cables back up, going through and changing all the BIOS settings, etc. Nothing major. The OS boot scripts have already been edited.

It's at 9 pm so I'm not driving 80 miles back home at the wee hours of the morning.

This is the last expected downtime for the forseeable future. Last week's performance problems have been fixed.
· · ·

23 Comments     (+0 »)
 
2007-12-03 5:29:40 PM  
... and I'm going to go try to power up the new board on the bench now so that I know it isn't DOA
 
2007-12-03 5:42:17 PM  
...aaaaand it's alive.
 
2007-12-03 5:43:37 PM  
30 minutes? That's pretty damn quick. Too bad it's when Heroes is on, we won't be able to talk about who we really want to die tonight and biatch when they off our favorite characters.
 
2007-12-03 5:51:48 PM  
BIOS settings done so I don't have to do that later.
 
ZAZ [TotalFark]
2007-12-03 5:57:46 PM  
...aaaaand it's alive.

5:42 PM, December 3, 2007. Fark 3.1 became self-aware. An army of time traveling assassins was unable to stop Mike. Humanity's extinction was inevitable.
 
2007-12-03 6:16:55 PM  
I think , Mike should get a round of applause for expediting the upgrades, and keeping us farkers happy, and generally kicking ass in the hardware dept.
 
2007-12-03 6:23:25 PM  
Mike 2.0 would have this accomplished in half the time.
 
2007-12-03 6:25:52 PM  
Yes, but Mike 2.0 is not RoHS compliant.
 
2007-12-03 6:27:57 PM  
Mike, it's been my experience that electronics work properly after a solid smack. Just try that.
 
2007-12-03 6:29:39 PM  
I'm only RoHS compliant if I've had enough sleep.
 
2007-12-03 6:51:51 PM  
whats a mutherbored...
 
2007-12-03 6:54:54 PM  
cazman: whats a mutherbored...

What mutha is when the kids are at skool and there's nothing on TV...
 
2007-12-03 7:13:10 PM  
How did you design the site's web architecture layout? Which loads are the load balancers handling? Are there cache servers? Is there a network file system in place for storing images? Did you replicate or cluster the db? Which storage engines did you go with (federated, innodb...)

The About Fark didnt give much info on what Fark's running on.

:)
 
2007-12-03 7:27:58 PM  
Hey mike...

Don't forget to ( insert easily forgettable thing ).

Steven

/(insert " i just wanna let you know we are all counting on you, good luck " )
 
2007-12-03 7:58:38 PM  
Mike, maybe you should press into service that fancy 1U Intel box you have... ;-)
 
2007-12-03 8:08:09 PM  
omega male:
How come you have a picture of Helene Bourgeois-Leclerc in your profile?
 
2007-12-03 8:19:22 PM  
oh, it's in service, just as a development box. Gonna work on it tonite too.

We may start a bit late. Got hungry. Stopped at DeSha's.

I was planning on getting into architecture details in a future blog post. Can't type that fast on this Treo...
 
2007-12-03 8:34:59 PM  
Remember the F5 party @ 9:30

img412.imageshack.usView Full Size
 
2007-12-03 9:50:37 PM  
All done now. Started late, but still did it in just under 30 mins
 
2007-12-04 1:00:49 AM  
Tu dou: How did you design the site's web architecture layout? Which loads are the load balancers handling? Are there cache servers? Is there a network file system in place for storing images? Did you replicate or cluster the db? Which storage engines did you go with (federated, innodb...)

The About Fark didnt give much info on what Fark's running on.

:)


That's kinda on purpose -- makes it harder for script kiddies.

Short(er) versions, backwards:

- InnoDB, with some MyISAM for tables (small ones mostly) that are generally readonly

- Just one DB box -- obviously a pretty beefy SCSI-based one (especially after tonight's mobo upgrade). This single point of failure is one I definitely need to address this year, along with maybe finally testing Postgres. There are actually a few replication slaves but they're for hot backups only (one's offsite) or the dev box and production webservers don't query them

- local copies of html/images instead of one NFS copy. Disk is cheap, html changes only every few minutes, images far less than that, and rsync (over NFS) works well enough for now. NFS performance tuning's kind of a black art...

- no Squid. Static content gets served from memory by way of the box having a crapload of memory and letting the OS's disk cache do the grunt work (which would not happen w/ NFS), plus all the webservers got memory upgrades this week. Local disk tends to get hit mostly for writing logs.

- it doesn't look like it from the way the site behaves, but all 5 webservers are exactly identical (including the hardware), and they all handle all the different hostnames. So "the forum servers are slow" doesn't actually make sense as forums.fark.com is the same 5 boxes that are handling www.fark.com and img.fark.com and everything else at full speed even when the forums are sucking mud. (Separate Apache/lighttpd instances bound to separate IP's on the same box; some have more/less modules enabled than others; saves tons of memory vs a monolithic config.) It just means they're sucking mud for some reason other than CPU :)
This is a little different than most people where they have one farm of img servers, one farm of cgi servers, etc. There's pros and cons to each approach obviously

- I'm leaving out hardware details (for now) because I farking hate Intel vs AMD fanboy flamewars. (Or PC vs Mac, or Windows vs Linux, or ATI vs nVidia...)

- I am hoping to merge everything (except images) down to just www.fark.com and (www.totalfark.com) by frontending everything with a proxy. I've got some other projects higher in the queue than that. I hoped the load balancers (Foundry) could have done it but they implement it a bit differently than I'd like and it requires turning off another feature I need.
 
2007-12-04 1:41:34 AM  
Mike: I'm leaving out hardware details (for now) because Intel vs AMD fanboy flamewars. (Or PC vs Mac, or Windows vs Linux, or ATI vs nVidia...)


Running apache, with perl? Linux hands down... And my hat is off to you sir. Gotta love the scalability, could you have expanded Fark to this point w/ windows server?

And Intel vs AMD I'm assuming you chose Intel b/c of the prevalence of hardware... seems to make things simple. I do detest AMD but sometimes things just dont work out the way they should. I mean, of course you can make AMD work in a server architecture but imho it is more suited to high power desktop apps...

And ATI vs nVidia... who gives a fark?
 
2007-12-04 2:00:31 AM  
We run FreeBSD.
 
2007-12-05 1:06:42 AM  
Do you partition your tables on category/link type?

I find its a pain in the butt handling db consistency when you design for scalability and have to drop fkeys. When I have to drop fkeys for partitioning/splitting up the db I try to use an Index table that keeps an updated copy of all denormalized fields so any non consistency can be checked against it and fixed.

Going to 64-bit Intel based servers with 64-bit mysql rocks. For handling the db server's single point of failure are you going to add a 2nd master and put it on a heartbeat or code a slave to take over as master?

If the social networking aspect of the site picked up heavily will things be able to cope with the onslaught? Facebook uses alot of akamai servers and Hyves.com leases over 200 servers and rents tonnes of rack space. It all starts with adding pictures to user profiles... :D

How are your server's work loads? Are you using/have you considered using virtualization to use up more of the CPUs?
 
Displayed 23 of 23 comments


This thread is closed to new comments.

Continue Farking




On Twitter


  1. Links are submitted by members of the Fark community.

  2. When community members submit a link, they also write a custom headline for the story.

  3. Other Farkers comment on the links. This is the number of comments. Click here to read them.

  4. Click here to submit a link.