If you can read this, either the style sheet didn't load or you have an older browser that doesn't support style sheets. Try clearing your browser cache and refreshing the page.
Posted by Mike at 2007-11-27 1:29:47 AM, edited 2007-11-30 4:29:22 PM (31 comments) | Permalink
• • •
7 clicks; posted to Main » on 27 Nov 2007 at 1:29 AM (15 years ago) | | share:
Well, this hasn't been the best couple of days. Performance has been t3h suck today, due to one or two separate issues combining and then causing others.
The short version is the images web servers got hammered by all the new images in user profiles and the submit script got hammered by a spammer all at the same time, which ran the boxes out of memory, which made them accept data from the database too slow, when then ran that box out of connection handles... so a nice little domino effect there.
Fixes so far:
- ordered more RAM, should be here by the end of the week
- removed the site logo images from the user profiles (left the topic tags in though)
- switched img.fark.com from Apache to lighttpd
- raised # of database connection handles; hopefully we'll be getting a new database box in January as well (after taxes)
- added about a half-dozen anti-spam features that I won't get into in case the spammers are reading this. More are forthcoming. (And if the stupid pigfarkers are reading this: submitting to Fark does nothing for your Google PageRank and will get you zero traffic, because it's not like we're going to greenlight it and it's not like Totalfarkers will click it, so why bother)
- fixed several slow SQL queries that were deadlocking other queries, and redesigned some table schemas to try to fix others
- made a config tweak on the load balancers
- recompiled database to be sure it's using the correct threading library
- Replaced database server's motherboard with a slightly faster temporary one (not as fast as I'd like but it's all I had sitting around); this one's capable of 64-bit. An even faster 64-bit board has been ordered.
- fixed several more bad SQL queries, two of which I think were the root cause of all this crap. Not that all the other issues didn't also need fixing, they just weren't as apparent til now...
- Still in progress: upgrading database server to a 64-bit OS and (once it arrives) upgrading its motherboard again
For more updates see the comments for this blog entry.
Side note: if you're using the FarkIt extension and you haven't upgraded to at least version 2.3f here, do it ASAP or you're likely to hit one of the anti-spam features.
· · ·
This thread is closed to new comments.