Managing Site Growth? 37
markmcb asks: "I started a web site about two years ago. When it began it was simple. The code was 75% hacked, and administration was easy. However, the times they are a changin'. Now I get hundreds of thousands of hits and have a steady flow of new users. I'm noticing an ever-increasing gap in terms of my site's popularity and its technological progression. Specifically, I have all sorts of 'XYZ for Beginners' books that are no longer of use to me. Even the so-called non-beginner publications seem to only scratch the surface of running a site. As problems get more complex, trying to Google every situation/issue I have with site administration has become less useful as well. I'm finding things like writing optimal code, configuring servers for high-volume, balancing ad income vs. server costs, and maximizing the efficiency from my moderation team have all become issues and that aren't addressed most books. What is the best way for a low-income, non-professional, but enthusiastic web designer/administrator like myself to manage site growth as it leaves the realm of just-for-fun?"
Hire a professional (or become one) (Score:0, Informative)
"Scalable" and "customized" are two things that when put together simply require a professional. And quite a lot of people calling themselves "professional" can't handle it, either.
Now, by "professional" I don't necessarily mean a degreed guy who makes at least $X thousand a year with Y years of experience. What I mean is, you're stepping into the domain where you can't hardly acquire the experience and skills necessary with anything less than full dedication usually brought on by having a job in the relevant domain.
There is, however, one other possibility for you to consider. If you analyze your needs and the available packages for your type of website, you may find that you can drop the "customized" aspect of it, if you can find a project close enough to your needs to require only minimal customization, perhaps even no actual code customization. Then you just need to import the data, and you will presumably have satisfied yourself that this package can meet your performance needs.
If the website you are referring to is the "OmniNerd" site you have a link to, then I would imagine this should be feasible. There are a lot of "news" packages, free and otherwise, and at least on first blush I don't see anything particularly unique about it. It looks an awful lot like slash, although from what I've heard that's not the easiest thing to customize. (slash hackers feel free to comment.)
Really, there's no excuse nowadays to start a new web framework from scratch, and your first impulse if your hack-job is starting to come apart at the seams should be to change to one of the umpteen bajillion tested, performant frameworks. Depending on your skill levels, which you did anything but talk up, you may even be missing basic pieces like caching, which is pretty important on a site like that. Non-professionals should not attempt to write website caching routines. Unless you want to go insane. (It's not that it's hard to write... it's that it's hard to get correct, and debugging cache problems are close to sheer hell.)
Re:Hire a professional (or become one) (Score:3, Informative)
Interesting question (Score:3, Informative)
I run a community website [debian-adm...ration.org] which is written in Perl with a MySQL back end.
Despite having just under 5000 users I had 3million hits last month, and shifted 13 Gb of traffic. Not bad for a single (dedicated) host!
There are two things that I'd suggest above all:
I use Danga's memcached [danga.com] which has a perl interface, but there are PHP ones too. This allows me to sensibly cache database queries (don't forget to test things to make sure you expire the cache appropriately!)
A combination of minimising queries and caching has kept me going even under a slashdotting.
If you have written the site code yourself I'd urge you to add a test suite. My site runs a full test suite every day [debian-adm...ration.org], and I run it manually whenever I make changes - this allows me to be sure that I'm not breaking things when I make changes.
Of course the standard development model of having a "live" site and a "test" site help here too. I develop the code on a laptop and store it under version control (CVS in my case, but it doesn't matter which system you use as long as you pick one) and only when it has passed the test suite do I push it to the live site.
Adding extra hardware can be an option for bigger sites, but I'm not at that point now. I had my biggest strain when the site reached around 1000 users, since then things keep ticking over nicely, and although it is growing it isn't growing terribly quickly which suits me fine. (There are a lot of users who visit the site via google searches and never register/return; I'd like to fix that, but I don't mind too much!)
Re:8... 8... 8 jobs in one (Score:1, Informative)
Programmer, Tester, DBA, System Admin, Application Admin, Content Admin, and User Wrangler. There's a lot to keep track of, and that's without managing the business side of it.
As for things I've learned as part of a team managing a large site:
(Note, we're J2EE, with a sizeable budget, so not all applies)
Get a good load balancing solution in place early. Get a strategy for where and when to add servers. Assume you will need to scale by adding servers, and not by replacing servers. Be prepared for asymetric load balancing. The server you buy two years from now may not be the same model you bought this year.
Caching is good. If you can cache a database driven page, even for an hour, do it. We use a combination of caching services and a massive prerendered static cache of our content. It keeps us running on a small number of web servers even when we are serving hundreds of millions of people a day. Caching page components, DB queries, and portlets. You get the idea.
Understand the performance profile of your code early and often. Keep some basic harnesses for specific compoents around to isolate performance bottlenecks. Use loading software. Does your code perform just as fast after 100 users have performed their action sequentially? 100,000? Are you loading configuration information from DB or disk for every new session that could be kept resident in memory? Are your DB indexes built correctly? Are you keeping the right amount of information in session? Are you logging too much information? Understand where in the system the bottleneck is for a component. Is it disk i/o? network? CPU? not enough RAM?
Optimization does turn into a game of diminishing returns. 95% or more of the performance gains you will realize will be in the top 5 code and configuration fixes. Far better to architect properly in the first place and create a system that can scale.
Ultimately, you'll need to start delegating eventually. There's far too much to do to manage a truly massive website alone. Start picking your focus, and if you can start delegating out some basic work to Rent a Coder, or a similar part-time operation, you may be able to keep yourself in good shape.