1) Market Research
I made this mistake too, once. I wanted to do some web SaaS stuff, had some ideas, which some test audiences told me were great, but when it came down to competition and pricing, I figured out that while my solution was technically superior and offered a lot of features that were lacking from other products, there was only a sub-set of users who actually cared. Of those, many of them weren't very technology oriented and I had trouble marketing the additional features. By comparison my competitors were offering less stuff for a much lower price and instead of improving their product, instead improved their pitch and sales force. I ended up abandoning that project and re-focusing my business on other services. It taught me that before jumping in and devoting resources to a project I better do an assload of research about the current state of the market, products, and customers. If you don't you spend a lot of time making no money.
2) Have a backup plan
Always have another option waiting for you. If it turns out your idea doesn't pan out, you can quickly refocus. It might be that the backup plan is going back to the job you were doing before, if you left on good terms; but you need to have your fingers in many pies as a small business.
3) Keep your skills/product fresh
Earthlink - in decline because modem users are drying up
RIM - Blackberries have looked pretty much the same for how many years? Even though other handsets offered significantly better internet experiences and had far more applications, they stuck with their text based email stuff. How's that looking for them now?
Novell - now dead, but was in decline for many years being supported by Groupware revenue. Without a new and compelling product to keep customers coming back, well...
Palm - They rode the PalmOS horse for many years, and didn't make very many changes to it, WebOS was too little, too late, they had already lost most of the market share
Business is all about evolution and change based on market demand and technology. If you're stagnant, you might be OK for a while, but eventually you're going to be out of date and after you loose the legacy customers that stay with you, you're sunk.
4) Know when to throw in the towel
One of the concepts in economics is Opportunity Cost, if you make $85K a year as an employee somewhere, but you're only making $40K a year as a small business, it's costing you $45K a year to operate your business! That might be fine for the first year or so, but if you're not seeing significant earnings or growth, just give it up (unless other intangibles like being your own boss and choosing your projects are worth the $ differential)
5) Project Growth Potential
There's been a shift in the industry my business services recently, there is a flood of cheap competitors from international sources that have been permitted into the market, which is creating price pressure on my products/services. These new competitors certainly don't have the expertise or provide a premium experience like I do, but a lot of consumers don't care, they make their buying decision based on price. I'm thinking that in 2-3 years I'll be making 1/2 of what I am now, which makes it just not worth it. But without constantly assessing the state of the market you're in, you're not making very good business and life decisions.