Skip to content
Do you have adblock enabled?
 
If you can read this, either the style sheet didn't load or you have an older browser that doesn't support style sheets. Try clearing your browser cache and refreshing the page.
Performance problems
Posted by Mike at 2007-11-27 1:29:47 AM, edited 2007-11-30 4:29:22 PM (31 comments) | Permalink
More: FarkBlog

•       •       •

7 clicks; posted to Main » on 27 Nov 2007 at 1:29 AM (15 years ago)   |   Favorite    |   share:  Share on Twitter share via Email Share on Facebook



Well, this hasn't been the best couple of days. Performance has been t3h suck today, due to one or two separate issues combining and then causing others.

The short version is the images web servers got hammered by all the new images in user profiles and the submit script got hammered by a spammer all at the same time, which ran the boxes out of memory, which made them accept data from the database too slow, when then ran that box out of connection handles... so a nice little domino effect there.

Fixes so far:

- ordered more RAM, should be here by the end of the week

- removed the site logo images from the user profiles (left the topic tags in though)

- switched img.fark.com from Apache to lighttpd

- raised # of database connection handles; hopefully we'll be getting a new database box in January as well (after taxes)

- added about a half-dozen anti-spam features that I won't get into in case the spammers are reading this. More are forthcoming. (And if the stupid pigfarkers are reading this: submitting to Fark does nothing for your Google PageRank and will get you zero traffic, because it's not like we're going to greenlight it and it's not like Totalfarkers will click it, so why bother)

- fixed several slow SQL queries that were deadlocking other queries, and redesigned some table schemas to try to fix others

- made a config tweak on the load balancers

- recompiled database to be sure it's using the correct threading library

- Replaced database server's motherboard with a slightly faster temporary one (not as fast as I'd like but it's all I had sitting around); this one's capable of 64-bit. An even faster 64-bit board has been ordered.

- fixed several more bad SQL queries, two of which I think were the root cause of all this crap. Not that all the other issues didn't also need fixing, they just weren't as apparent til now...

- Still in progress: upgrading database server to a 64-bit OS and (once it arrives) upgrading its motherboard again

For more updates see the comments for this blog entry.

---

Side note: if you're using the FarkIt extension and you haven't upgraded to at least version 2.3f here, do it ASAP or you're likely to hit one of the anti-spam features.
· · ·

(view entire blog)
31 Comments     (+0 »)
 
RocketRod
2007-11-27 4:47:18 PM  
Well, keep at it. I know it sucks, but from what I can see... it's pretty good so far.
 
Mike [TotalFark] [OhFark]
2007-11-27 5:56:23 PM  
It's getting better but not quite there yet. Found and fixed some more software problems today.
 
40below [TotalFark]
2007-11-28 1:21:32 PM  
I like having the source tags turned off in the profiles, it looked pretty cluttered before.
 
Mike [TotalFark] [OhFark]
2007-11-28 4:22:48 PM  
That was actually done to cut load on img.fark.com. :)

The load problems are much better today but not completely gone. I still haven't found the root cause. On the other hand, it's not like all these other performance tweaks are exactly harmful -- they needed doing anyway. Especially switching img.fark.com over to lighttpd -- it's impressively fast :)
 
realwx
2007-11-28 5:54:54 PM  
I've noticed today that the forums.fark.com part were in and out, including when I've looked in here or other threads and found my browser still stalling. It's gone now, though...

/btw thanks Mike for putting in the "Preview comments before posting" option in Preferences
 
Mike [TotalFark] [OhFark]
2007-11-28 6:03:30 PM  
That option has been in profiles for several years. Given all the questions about flipping the preview box around, it was kinda obvious that nobody knew that :)
 
paleryder69 [OhFark]
2007-11-29 12:43:07 PM  
still kinda twichy today. I have had several posts time out.
 
helchose
2007-11-29 12:50:36 PM  
Mike: That option has been in profiles for several years. Given all the questions about flipping the preview box around, it was kinda obvious that nobody knew that :)

I noticed it. As long as the tab sequence stays the same, I have no issues.
 
SoonerGrad
2007-11-29 1:16:51 PM  
I say turn off the source images in the profiles.
 
keithgabryelski [BareFark]
2007-11-29 1:16:57 PM  
Today it seems like there is a longer (wishful thinking?) timeout before receiving a browser error "Problem loading" instead of the Five Oh Oh fark page.

Is the new delay a win (that is, will I ever receive the page queried if it takes more than 10 seconds to fetch the page?)

I'd rather see the 500 page.
 
DaCricket [TotalFark]
2007-11-29 1:28:26 PM  
Mike: That was actually done to cut load on img.fark.com. :)

*snork*

/I knew it. (pops)
 
timmy_the_tooth
2007-11-29 1:28:27 PM  
paleryder69: still kinda twichy today. I have had several posts time out.

really? i can't hardly see posts -- much less comment.

i'd say "completely twitchy"
 
fourply
2007-11-29 1:29:45 PM  
bolg!
 
AlanSmithee
2007-11-29 1:49:27 PM  
I'm still timing out for hours on end when it comes to accessing the forum threads.
 
winterlongone
2007-11-29 1:53:13 PM  
well, mike, feel vaguely comforted by the fact that the oral surgeon has left me miserable with another visit i have to face next tuesday. i suspect your pain level right now is approximate to mine.

/yours may be worse
//on the other hand...the damn oral surgeon won't Rx good pain meds, so i hate him right now....
 
keithgabryelski [BareFark]
2007-11-29 2:27:53 PM  
I take back my "after 10 seconds its wishful thinking" statement.
I just noticed my post go through will a near 30 second delay.
 
omega male
2007-11-29 5:45:57 PM  
helchose: As long as the tab sequence stays the same, I have no issues.

that
 
Mike [TotalFark] [OhFark]
2007-11-29 8:18:05 PM  
Memory arrived today and all webservers have been upgraded.

There are still some CPU issues on the database server that I can't pin down to any specific queries.

I've just replaced the motherboard with a newer one -- from a Dual 2.66 ghz Xeon (Prestonia) to a single 3.0 ghz Pentium D (Presler). It's not the board I wanted but it's all I had sitting around that had a PCI-X slot for the RAID card. (It's actually the Asus P5WDG2-WS board from my old gaming rig) :)

One thing it will let me do is switch to a 64-bit OS (the Prestonia Xeons are 32-bit only) which will make a huge difference in how efficiently 4GB of memory gets used. Anyway, the Presler has quadruple the L2 cache that the Prestonia does so it can't exactly hurt things -- except that the onboard NIC sucks.

I'll see how tomorrow goes and if it helped any, I'll get a cheap Core 2 Quad board for it. What I really want is a dual Harpertown system (with SAS RAID instead of the current U320 SCSI) but they aren't exactly shipping in volume yet... so... that's January. Once we do get that though, I can use the Core 2 Quad for one of my two personal servers, or better yet, for a slave database box so we can actually have some failover on that finally.

Meanwhile, I'm still trying to track down why the CPU was getting pegged in the first place. It seems to be lock contention and it might be down in the OS kernel -- still investigating. The web servers should be in good shape now, at least -- having them swapping a bunch on Monday wasn't helping and now they should never swap.
 
AlanSmithee
2007-11-29 9:19:06 PM  
Things running smoothly now. Tnx
 
realwx
2007-11-29 9:21:48 PM  
AlanSmithee: Things running smoothly now. Tnx

Same here. No outages ever since Mike updated the servers.
 
adragontattoo
2007-11-29 10:17:13 PM  
Mike: Memory arrived today and all webservers have been upgraded.

There are still some CPU issues on the database server that I can't pin down to any specific queries.

I've just replaced the motherboard with a newer one -- from a Dual 2.66 ghz Xeon (Prestonia) to a single 3.0 ghz Pentium D (Presler). It's not the board I wanted but it's all I had sitting around that had a PCI-X slot for the RAID card. (It's actually the Asus P5WDG2-WS board from my old gaming rig) :)

One thing it will let me do is switch to a 64-bit OS (the Prestonia Xeons are 32-bit only) which will make a huge difference in how efficiently 4GB of memory gets used. Anyway, the Presler has quadruple the L2 cache that the Prestonia does so it can't exactly hurt things -- except that the onboard NIC sucks.

I'll see how tomorrow goes and if it helped any, I'll get a cheap Core 2 Quad board for it. What I really want is a dual Harpertown system (with SAS RAID instead of the current U320 SCSI) but they aren't exactly shipping in volume yet... so... that's January. Once we do get that though, I can use the Core 2 Quad for one of my two personal servers, or better yet, for a slave database box so we can actually have some failover on that finally.

Meanwhile, I'm still trying to track down why the CPU was getting pegged in the first place. It seems to be lock contention and it might be down in the OS kernel -- still investigating. The web servers should be in good shape now, at least -- having them swapping a bunch on Monday wasn't helping and now they should never swap.

Yeah last I knew, NO 32 bit OS was effectively able to use over 3gb of Ram without some interesting side effects.

I have had machines that for some reason got pegged and it ended up being an OS issue, a fresh install and there was no more problems, might want to try that Mike in case you care/wonder/read this hehe.
 
Mike [TotalFark] [OhFark]
2007-11-30 2:33:24 AM  
OK. I'm still seeing CPU spikes every half hour, they're just less severe, and because it's overnight, the CGI's are slowing down any from it. And now I've pinpointed at least some specific program that's causing it, and it's because it's using the wrong index on a query table. So I'm gonna try to fix that before going to sleep.

Going from 32-bit to 64-bit FreeBSD is pretty much a fresh reinstall anyway, or at least "recompile everything including the userland OS, the kernel, and all the installed apps". It took about half a day the last 8 times I did it. The good news is this is the next-to-last 32 bit box we have.
 
Mike [TotalFark] [OhFark]
2007-11-30 2:34:00 AM  
(on a "query table?" English, dumbass. table query. It's late.)
 
Mike [TotalFark] [OhFark]
2007-11-30 2:35:21 AM  
cgi's *arent* slowing do... ya know, screw it, when English fails it's time for sleep. :)

80 mile road trip each way to/from the data center plus a stopoff at a LAN party on the way home...
 
ahab
2007-11-30 8:50:43 AM  
Mike: 80 mile road trip each way to/from the data center plus a stopoff at a LAN party on the way home...

We appreciate the work though (or at least, we'll pretend to appreciate it until the first sign of a problem, then we'll submit 12837189 threads about how slow Fark is and what a ripoff it is). :)
 
Paryb
2007-11-30 1:47:18 PM  
Mike, I just want to tell you both good luck. We're all counting on you.
 
Mike [TotalFark] [OhFark]
2007-11-30 2:45:16 PM  
Right. Now I'm hitting 32-bit limits again -- on the old board it reserved about 256K out of the 4GB for PCI space, this new board is reserving a whole 1GB. So the db server is running out of memory and actually crashing/dumping core even though there's plenty of physical memory left. So first priority is to get to 64-bit ASAP. That'll involve a little more downtime, but at this point, is anyone really gonna notice? :p

I did order a new motherboard on my own dime for now. I doubt it'll make it here by tomorrow even w/ FedEx overnight. Nobody local stocks good server-class motherboards (I tried several stores yesterday).

If there are any MySQL gurus in the house, a pointer to a tool that can take an all-query log and replay it with the same timing and concurrency as the original, would help me bash on a test copy to try to reproduce app-level problems... as opposed to just running every query in the log one at a time in sequence... I don't think supersmack can do that...

Meanwhile, I did fix the one app that wasn't using indexes right.
 
Mike [TotalFark] [OhFark]
2007-11-30 4:16:23 PM  
Found another bad query, fixed it
 
Mike [TotalFark] [OhFark]
2007-11-30 4:35:15 PM  
...and another, and I think those two were the root cause of the problem.
 
realwx
2007-11-30 7:42:33 PM  
By the way Mike, you may be interested in this, if people use Firefox. It could be used for some tags instead of finding ways to make them not drain your processes.
 
paleryder69 [OhFark]
2007-12-01 1:21:28 AM  
Mike,

I am not sure exactly what your are looking for however googling this term "mysql query log analyzer" brought up several promising tools.

If you need someone to do some grunt work for you just let me know. I got the time to spare. I am not an expert in *nix but I have worked with both and did intermediate TS for some programmers in a production environment.
 
Displayed 31 of 31 comments


Redisplay/refresh comments
This thread is closed to new comments.

Continue Farking

If you like these links, you'll love

TotalFark

The next best thing to UltraFark

learn more | sign up
On Twitter

  1. Links are submitted by members of the Fark community.

  2. When community members submit a link, they also write a custom headline for the story.

  3. Other Farkers comment on the links. This is the number of comments. Click here to read them.

  4. Click here to submit a link.