Most of it is the way it is because it evolved that way, and because of the laws/rules under which it all evolved. You paint with too broad a brush when you say that the vast majority of today's financial instruments have been created out of thin air. That's nonsense spoken out of ignorance, the same way a non-geek might say, "why can't software designers create programs without bugs?"
Commercial paper is a very broad term and encompasses everything from promissory notes to normal consumer checks. Just about any transaction not involving cash or electronic transfer is done with commercial paper. A huge portion of financial transactions are still done with commercial paper. So in the general sense of the term, it is still very, very necessary.
Now if you want to start examining specific financial instruments, like the derivatives backed by (partially) crap mortgages, we can have a conversation. I think the idea behind those instruments was basically sound, but the things ended up being a lot more complicated than people thought. It makes sense that if you lump a bunch of mortgages together, only a small percentage of those will default, thereby distributing your risk. But in a climate where fraud was rampant and the people signing people up for mortgages had no incentive to make sure people could actually pay those mortgages back, your lump of mortgages has a much higher chance of containing too many bad mortgages to make the resulting instrument profitable.
The derivatives market had the perverse effect of creating and encouraging that climate, because the mortgage buyers would buy without enough questions because they knew there were buyers who would buy the derivatives without too many questions. The fundamental problem with the whole concept, it seems to me, is that the derivative buyers and sellers forgot to insist on and question the credentials of the individual mortgagees they were investing in. Had they done a little bit of verification there, we might not be in this place right now.
