Well firstly, I haven't worked in areas actually responsible for making these decisions. The closest I've done is mortgage fraud - it's actually quite complex drawing the line between a wilful fraud on the customer's part and a mistake on the bank's, and it can make a big difference to who foots the bill. However, to give the best answer I can: no, no one is really shocked because most people either knew it or take the view that something always bursts the bubble. It's weird actually, I was working with a researcher at the University of Cambridge (England) analysing the dot-com bust in comparison to other busts (all the way back to the tulip bubble in the 1600s). In the end, I couldn't get the stock market data we had to show the pattern of decline she was expecting so we came up with an alternative explanation. A couple of years later the explanation seems obvious - the decline that should have occurred was overtaken by a property boom fuelling credit-driven growth. N.B. I'm not saying we saw this coming, our explanation was much more mundane and convincing than "it's all going to explode in two years' time".
Back to the point in hand, I think the reason people went ahead and did it anyway is a combination of 1) division of labour: each person just hits their own targets, and leaves the bigger picture to 'management' who are really just hitting their own targets. By this account, it's ultimately governments' fault for not reigning in the bubble as they have cushioned the bust. 2) Judging sentiment: the reality is that, if you are clever but maybe a bit naÃve, the bubble doesn't really make sense because the economics are wrong from the start. But any good trader (and any bad one for that matter), knows that as long as everyone else is 'stupid', there's only one way to make money: follow the herd. So there's no point fighting it - just get out first. Goldmans are a good example here. They have been screwed by the change in sentiment, not the economics - they are and always have been making money, and hedged their way out of sub-prime before most others saw it coming. 3) Hidden feedback loops, like AIG effectively speculating through certain complex types of insurance contract, the hedge-fund connections with lenders or prime brokers who go bust, Fannie Mae and Freddie Mac lending to one another (that one I find really unbelievable - it's the financial equivalent of passing the bottom of a ladder up to the top in order to climb higher, you just have to keep going faster and faster as the fall accelerates; the reason it was missed is that, from a distance, people on the ladder look stationary (okay, I stretched that analogy too far)). That last point isn't so much about why the lending happened, more about why people assumed it wouldn't be so bad if it went wrong.
The overarching point is partly one of greed, but also one of entropy, or what I think of as the greed-entropy cycle. The basic tenet is a response to your implication that "someone should have seen this coming". Yes, someone did see this coming, and some people responded, but then others saw that response coming, and responded, etc. All this layers up complexity very quickly. One doesn't really pick this up in the popular press, but having been, as I say, inside some of these organisations, I can attest that most of them don't even understand what's going on in their own company (and I'm talking about multi-million-dollar 'holes' here, like the Lehman-UBS-Olivant one, which only got noticed because Lehman went bust). So you can imagine, it's really a bit of a stretch to claim anyone should understand the whole system. Anyway, if they could model it (e.g. a clever hedge fund with millions of pounds of computer), the temptation to use that knowledge for profit would likely overcome them, and the moment such a model began to trade with the existing system, the system would effectively double in complexity and become unpredictable again.
