As someone who has written a couple of novels and has them available both on Amazon and as free downloads...you're absolutely correct.
I get a certain amount of downloads of the free copies of my novel every month; I can't say for certain that real people are downloading them as opposed to search engines or bots, since my stats-tracking software (awstats) isn't really that specific and I'm too busy to bother poring through apache logs, but I'm guessing it's the latter. I've had one of my novels available on Amazon since late July, and I've sold a total of 2 copies, both to people I know. I've tried three different price points, and followed all of the "recommended" advice in terms of twitter, Goodreads, blogging, etc., and this has had zero impact on sales.
The short of it is, as an unknown author, people simply won't take the "risk" of spending the time on my novels even at a price of "free." And to be honest, I don't really blame them. I'd have a hard time justifying reading something from an unknown author myself, without some body of reviews or high-profile recommendations to fall back on. This is, of course, a giant catch-22: you can't really get non-purchasers or non-readers to review books they haven't read, it's hard to find reviewers willing to review your work if you are unknown, and readers won't buy your unreviewed novel (or even bother to read it for free), much less contribute their own reviews. I think it also doesn't help that I write stories in genres (primarily SFF) that are not thrillers or romance; the readership in those genres seems to be a bit less selective and more willing to take chances (or perhaps have more time to risk, I'm not entirely sure), if anecdotes from authors like John Locke and Amanda Hocking are any indication.
That said, I still don't feel any desire to cave and go the traditional publishing route. From all the information I can gather, you're just as unlikely to succeed there, and luck (and your own unassisted self-promotion) will play just as big a part in your results as if you just bypass the existing publishing industry completely.
Side note: for readers who don't know, Amazon uses a tiered royalty system where you can only get 70% royalties if your book is priced no lower than $2.99 USD. If you price it at $0.99 USD, you will get approximately $0.35 for each copy you sell, minus a bandwidth charge dependent on the size of your book. The price point seems to be irrelevant in your book's success, as far as I can tell; I've sold no copies at $0.99, $2.99, or $4.99 other than the two copies I initially sold to a couple of people in my immediate social network.