The reason the cgi URLs all had /cgi/fark dates way back 10 years to a long-gone cgiwrap config, when Fark was sharing a single machine with about 200 other websites. The need for that disappeared when we got our own hardware, which was in about, uh, 2001 or so... At the time I didn't fix it because I wasn't very good with mod_rewrite, or for that matter regexps in general. That's not the case anymore, so, we're fixing it. :)

As for the different hostnames, that's a bit harder...

They aren't actually different machines, or even clusters of machines, they're separate Apache instances running on different IPs on the same set of machines. (Some people would run them on different ports on the same IP but for various reasons, we didn't do it that way.) The reason they're split up is because mod_perl, which handles all the cgi scripts, uses a lot of memory to get its performance by keeping all the scripts pre-compiled in memory at all times, so there's no fork/compile overhead on each hit. But the front page of Fark, and all the front tab pages, are all static files and not scripts run on the fly -- that's the #1 reason we can run on such minimalist hardware. So www.fark.com runs a stripped-down Apache instance with mod_perl removed, which of course cuts memory overhead way down. img.fark.com is stripped down even more than that... and in fact last year we switched the img.fark.com vhost from Apache to lighttpd to get it streamlined even more. The cgi.fark.com vhost was getting kinda memory-bloaty, so we split the higher-volume go.pl and comments.pl stuff out to two more Apache instances with just those scripts preloaded.

But having all these hostnames makes the site look ugly and it makes stats difficult... and having the front page static severely limits possible features -- for example, it's why we can't have any per-user customizations for the front page of Fark. There's a hell of a lot of cool things we could do if we had that, but we can't exactly have an ugly cgi.fark.com/cgi/fark/* URL mess for our main page either. And for performance reasons it'd be nice to keep serving the static one if the user isn't logged in. Merging all the configs back into one isn't real practical either -- sure, memory and CPU are much cheaper these days, but they're not free :)

So the easy solution is to just stick a reverse proxy on the front of everything, with a bunch of URL rewriting rules; that way, you can have all the microoptimization of multiple Apache configs yet still have a unified looking front end. It's a pretty common setup for sites running things like Rails.

After researching the tons of options available for doing that -- using our existing load balancers' URL switching features, using Apache's existing mod_proxy (which I use in production on another site), our current lighttpd, or other things like nginx, Squid, Varnish, haproxy, pound, perlbal, etc etc... after a few days my head started to hurt and I finally decided to start playing with nginx. Turns out it benchmarked faster than lighttpd for just image hosting too, which in turn was faster than Apache (prefork mpm, anyway; worker mpm's likely comparable)

So that's the new architecture... nginx in front serving as much static content as possible, existing Apache behind it for dynamic content, and rewrite rules to transparently hide the Apache mess. I'm already running foobies.com and our images in production this way, and the setup for www.fark.com is reasonably well tested and ready to roll... but as I've got plans this weekend that involve a lot of alcohol, I'd rather not roll it out until next week just in case something goes unexpectedly blooey.

Some of what Farky does couldn't be built in at our end because of the whole static-front-page thing, which prevents per-user front page customizations. Fixing that and allowing us to add that kind of functionality is part of why we're doing this.

So short term though, yes it likely will break, long term, it won't be needed as much anyway. I don't even know who maintains that thing anymore, but if they email me I can see about helping them write a patch.

Re-read the original blog post up to the words "all the old URLs will continue to work" and that's all you need to know in as simple terms as anyone can really put it.

Friday (Mar 6, tomorrow) 5 PM ET is when this switchover happens.

OK, that didn't go super smoothly, but it's done