Let's be clear here -- no currency is backed by anything, ever. Sure, "old currencies" were backed by a precious metal. What was that backed by? The answer to "why do currencies have value" is that _nobody knows_. The different schools of economics all have different theories, but macroeconomics is not, despite what people say, a science.
The theory backing bitcoins is largely based on the (non-mainstream) ideas of Austrian economics; which claim that currencies have value almost entirely because they are scarce, fungible, and useless. That is, their value as an exchange medium far exceeds their value as physical objects. Incidentally, Austrian economics is pretty much the only school of economics to openly acknowledge that it is not a scientific theory.
Fiat money qualifies easily -- nobody is burning dollars for heat or using them for wallpaper, and the scarcity is guaranteed by the issuing government (although easily abused, and sometimes catastrophically, which is why Austrian's tend not to like fiat currencies).
Gold qualifies with caveats; it's clearly "money", but it suffers from portability and verifiability problems unless it's minted into a form that is hard to reproduce (the old-old-school approach) or vouchers are issued that can be redeemed for stored gold (the newer-old-school approach). The vouchers then become money in their own right, since their scarcity is tied to the scarcity of the underlying metal, and they're just as fungible and useless. But almost universally, governments (and unscrupulous banks, and sometimes even scrupulous banks) abuse the fact that the holder of a voucher can't see the gold to steal the gold; effectively disconnecting the scarcity of the two. Same goes for silver, though less dramatic.
It's surprisingly hard to come up with examples of things that are scarce, fungible, and useless that are have not been used as currencies. Even things that violate one of these things is often used as money in a barter sense or in the short term (i.e. cigarettes in prison, wampum among east-coast Native Americans, Facebook game fun-dollars, baseball cards, or the Iraqi Swiss Dinar).
So this is what Bitcoins are -- they are nothing but pure scarcity, fungibility, and uselessness. The portability is nice, except that transactions are tricky (since there's no real "receipt" mechanism -- verifying that a customer has paid his bill requires funky gymnastics), all use of it is dependent on the accessibility of the internet, and the scarcity is conditional on there being sufficient computing power applied to the problem. But aside from those, it really should work. And the fact that they are worth anything at all (that people are willing to buy them at any non-zero price) is a big deal.
In terms of price stability, this is a complicated phenomenon; Austrian's generally do not consider price increases to be the same as inflation (a purely semantic distinction). What causes price changes are complicated, and can not be reduced to a simple rule, because at any moment, technological progress and other myriad phenomenon are messing with prices that affect, to a small extent, everyone in the supply chain of every business in the world, and that effect get compounded over time. The most visible form of this is the supply chain of money itself, through the mechanisms of banking and credit.
So why is the Bitcoin value so volatile? (Now we're in the realm of pure speculation on my part) Because there's no supply chain to keep prices stable. No supply chain of any sort; a restaurant that accepts bitcoins can not buy silverware, or food, or paper cups, or cash register tape, or POS systems, or pay rent in Bitcoins. There are no mechanisms for loaning bitcoins; in either direction -- there are no banks that will pay you to lend them your bitcoins, and no banks that will assess your trustworthiness and income and lend you the money lent to them. So the price floats, while people try to figure out how to provide baseline services (like web hosting), so that they can tie their own supply chain to bitcoins. These rugged pioneers will undoubtedly be screwed royally -- their supply chains still require dollars, so they have to change their prices constantly to reflect the changing price of bitcoins in dollars; which means that services which use them can't rely on that price stability, and need to charge their clients more bitcoins.
So either the economy develops, and complete supply chains, including credit markets, evolve and have the effect of stabilizing the price, or the value drops to the point where the scarcity of the currency dissolves (that is, it becomes cheap to attack the system directly). But this story is far from written yet.
I'll close this rant with one more comment; the idea that money has value because it is used for taxation is laughable -- Euro's have value in the US despite the fact that the US will not accept taxes in them. You can run a business entirely on barter; the transfer to US dollars for tax purposes is based on the valuation of the barter goods in US dollars, so a 10% tax on dollars is the same as a 10% tax on sheep (or Euros, or Bitcoins). If the barter good becomes cheaper or more expensive (valued in US dollars) then so does the tax burden, in proportion to the change in price of the barter good.
