Managing Site Growth? 37
markmcb asks: "I started a web site about two years ago. When it began it was simple. The code was 75% hacked, and administration was easy. However, the times they are a changin'. Now I get hundreds of thousands of hits and have a steady flow of new users. I'm noticing an ever-increasing gap in terms of my site's popularity and its technological progression. Specifically, I have all sorts of 'XYZ for Beginners' books that are no longer of use to me. Even the so-called non-beginner publications seem to only scratch the surface of running a site. As problems get more complex, trying to Google every situation/issue I have with site administration has become less useful as well. I'm finding things like writing optimal code, configuring servers for high-volume, balancing ad income vs. server costs, and maximizing the efficiency from my moderation team have all become issues and that aren't addressed most books. What is the best way for a low-income, non-professional, but enthusiastic web designer/administrator like myself to manage site growth as it leaves the realm of just-for-fun?"
Re:Start charging for it. (Score:1)
As for the second part, as soon as you get bored of it, you should have already had a few zealous volunteer mods who would sell their toes (but not their typing fingers, of course) for the opportunity to enhance the XYZ technology for use on your site, and in return you could simply give them some somewhat restricted access to one of your servers so that they could st
Find a local nerdling (Score:2, Funny)
Re:Find a local nerdling (Score:2)
1.Offer him Coke and Candy
2. ???
#. Profit!
Your post wasn't clear.
Re:Find a local nerdling (Score:2)
Hire a professional (or become one) (Score:5, Insightful)
"Scalable" and "customized" are two things that when put together simply require a professional. And quite a lot of people calling themselves "professional" can't handle it, either.
Now, by "professional" I don't necessarily mean a degreed guy who makes at least $X thousand a year with Y years of experience. What I mean is, you're stepping into the domain where you can't hardly acquire the experience and skills necessary with anything less than full dedication usually brought on by having a job in the relevant domain.
There is, however, one other possibility for you to consider. If you analyze your needs and the available packages for your type of website, you may find that you can drop the "customized" aspect of it, if you can find a project close enough to your needs to require only minimal customization, perhaps even no actual code customization. Then you just need to import the data, and you will presumably have satisfied yourself that this package can meet your performance needs.
If the website you are referring to is the "OmniNerd" site you have a link to, then I would imagine this should be feasible. There are a lot of "news" packages, free and otherwise, and at least on first blush I don't see anything particularly unique about it. It looks an awful lot like slash, although from what I've heard that's not the easiest thing to customize. (slash hackers feel free to comment.)
Really, there's no excuse nowadays to start a new web framework from scratch, and your first impulse if your hack-job is starting to come apart at the seams should be to change to one of the umpteen bajillion tested, performant frameworks. Depending on your skill levels, which you did anything but talk up, you may even be missing basic pieces like caching, which is pretty important on a site like that. Non-professionals should not attempt to write website caching routines. Unless you want to go insane. (It's not that it's hard to write... it's that it's hard to get correct, and debugging cache problems are close to sheer hell.)
Re:Hire a professional (or become one) (Score:3, Informative)
Re:Hire a professional (or become one) (Score:1)
I am now in the proces of rolling our several more Plone sites on my own, with the help of a large and helpful Plone commmunity. [plone.org]
Designed to scale well, themable (CSS-based), and good caching options (squid and the new CacheFu [plone.org]).
I have found Plone to provide a great foundation to build upon.
lollerskates (Score:2, Funny)
use others experience (Score:3, Interesting)
I think it's very unlikely you will find books in that area considering that when you reach a certain level of complexity technology changes too fast to make a book relevant.
Re:use others experience (Score:4, Interesting)
Adding more hardware is always a way to dig yourself out of a digg, but be careful you don't just look at how the big boys do it and think that's right. Smart people can do more with less. Look to Python, Perl, C# (but not ASP.Net unless you really know what you're doing) who have mature libraries. PHP and Coldfusion suffer too much from the type of app that's built-up and destroyed on every page load. It doesn't encourage separation from what only need to be done once (app initialisation) and page response, which is a large part of optimisation (both in doing things every page load that should be done once, and seperating out the page data and having layers of cache from the DB and web-templating). Learn about http headers and SquidCache.
So far as software recommendations I'd say Python with CherryPy and Kid Templates (but not turbogears). It's fast and simple.
You know... (Score:2)
Maybe we need to have that discussion later... or tomorrow...
Re:You know... (Score:1)
Depends upon the platform (Score:4, Insightful)
Open source technology tends to be more scalable and solid, but even there, a bad choice stifles your progress. If money is no object, I guess you can always scale up, but the commercial platforms often have their admins spending more time patching and maintaining the status quo than progressing. The bigger question is: Did you do your homework when you initially designed the system? If you're stuck, that's likely the problem.
If you have a choice to redesign or redeploy your site, what you need to do is ask yourself, not whether or not the technology you're familiar with can do what you want, but instead, are you using the right technology to do what you want?
The Slashdot(TM) Method (Score:5, Funny)
2) Allow posting comments on your stories.
3) When people abuse comments, put in moderation system to stop them.
4) Hire some random writers with axes to grind, like "Geeks are oppressed by stupid conformist society".
5) When people abuse moderation points, throw in meta-moderation system to stop them.
6) Hype yourself up, claiming you are part of some revolution in media that will bring control to the masses.
7) Sell out to some venture capitalists.
8) Abuse moderation and metamoderation system yourself when comments piss you off.
9) Cover the site in more ads than Times Square.
10) Stop putting any effort into the site whatsoever.
11) Let people pimp their own blogs in story submissions.
12) Charge money to preview stories so people can read links before hordes of visitors take sites down, a problem you caused in the first place.
13) Charge money to view the site without ads.
14)???
15).... profit!
As you can see, the main goal of the Slashdot Method is to cause problems yourself, then charge people money to fix them. But remember, the most important step is to have fun! Hope you have as much success as they did!
Re:The Slashdot(TM) Method (Score:2)
8... 8... 8 jobs in one (Score:2)
For strategic planning, go to SCORE (a volunteer group of retired executives who advise small business owners) and get a couple of advisors... one to show you how to break down and analyze the financial numbers, one to help you learn how to better manage y
Re:8... 8... 8 jobs in one (Score:1, Informative)
Programmer, Tester, DBA, System Admin, Application Admin, Content Admin, and User Wrangler. There's a lot to keep track of, and that's without managing the business side of it.
As for things I've learned as part of a team managing a large site:
(Note, we're J2EE, with a sizeable budget, so not all applies)
Get a good load balancing solution in place early. Get a strategy for where and when to add servers. Assume you will need to scale by addi
Interesting question (Score:3, Informative)
I run a community website [debian-adm...ration.org] which is written in Perl with a MySQL back end.
Despite having just under 5000 users I had 3million hits last month, and shifted 13 Gb of traffic. Not bad for a single (dedicated) host!
There are two things that I'd suggest above all:
I use Danga's memcached [danga.com] which has a perl interface, but there are PHP ones too. This allows me to sensibly cache database queries (don't forget to test things to make sure you expire the cache appropriately!)
A combination of minimising queries and caching has kept me going even under a slashdotting.
If you have written the site code yourself I'd urge you to add a test suite. My site runs a full test suite every day [debian-adm...ration.org], and I run it manually whenever I make changes - this allows me to be sure that I'm not breaking things when I make changes.
Of course the standard development model of having a "live" site and a "test" site help here too. I develop the code on a laptop and store it under version control (CVS in my case, but it doesn't matter which system you use as long as you pick one) and only when it has passed the test suite do I push it to the live site.
Adding extra hardware can be an option for bigger sites, but I'm not at that point now. I had my biggest strain when the site reached around 1000 users, since then things keep ticking over nicely, and although it is growing it isn't growing terribly quickly which suits me fine. (There are a lot of users who visit the site via google searches and never register/return; I'd like to fix that, but I don't mind too much!)
Re:Interesting question (Score:1)
One thing which I just remembered - so I'll post it now even if it is a little late - the single biggest speedup/optimisation I made to my site was to disable DNS lookups in the Apache logfiles.
In the normal course of things this isn't a big deal, but try surving under a /. attack whilst you're getting a ton of incoming connections and your bandwidth is saturated - suddenly theres nothing "spare" to do the DNS lookups for logging purposes.
Nowadays I disable DNS lookups for the logfiles as a matter of poli
This is a great example of what acadmia teaches yo (Score:2, Insightful)
However, even from day one, I could see they were trying to teach us something else, and not just how to program. This wasnt clear or evident to anyone else who didnt know how to program in a compiled language, because the workload of simply und
I've learned stuff (Score:1)
Googl
Are you Tom Anderson? (Score:1)
What you really need... (Score:2)
Tips from the trenches (Score:3, Insightful)
Re:Tips from the trenches (Score:1)
And by the way, another tip, if the coder necessarily has to use PHP - try organizing the functions using OOP. In the end the bad organizing on the website staggers the coder because of the confusing construction.
Re:Tips from the trenches (Score:2)
Also, consider writing your own Apache module. Scripting languages are *slow*.
you can always give up (Score:3, Interesting)
The site had gotten so it wasn't fun for me. I was home every night after work writing PHP, and coming back from a weekend vacation was a nightmare. I also felt like all those members deserved more than I could give them.
Business professionals will tell you that it takes a certain type of person to get a business going, another type to get it stable, another type to get it to the top, etc. The web is the same way, and if you're a "starter" who can come up with innovative concepts that take off, then go do that. Don't get tied down with old projects.