Some advice for programmers trying to do interface design: Don't. Leave question like this to an interface designer. If you can't afford one, or you want to do it anway, a good book for starters is "The Design of Everyday Things" by Don Norman, it's not even expensive. Further, stop treating users as a problem in the system. Every user has his own model on how the system works. This model is very likely very different from the programmers model. Your task as an interface designer is to teach the user enough about the system (or it's model), so he can use the system successfully. Error messages don't help very much, as you've figured out, users don't read them. The lesson is, don't try to force the user to read error messages, instead find other ways to communicate the model. Often, it's a good idea to think about the problem in an abstract way. For example, we have a similar problem at the place I work. There are two doors next to each other, one you should use, the other one you mustn't because it triggers the alarm. They tried to fix it by attaching a sign saying not to use that door. Needless to say, it didn't work, because noone read the sign. Just like your error messages, this sign was completely ignored. It's not wrong of the users to ignore the sign, quite the opposite: We have to filter out information to survive. If you pass through your environment, you too ignore information, i. e. I don't think you read every sign in your proximity. I have no idea why they couldn't come up with a better solution for the door: Locking it would be very easy. Even better, by removing the door handles it would be very clear that the door can't be used.

One major problem of captchas is that usually blind users can't solve the captcha. So you effectively lock out disabled persons from your website, a fact that is rarely mentioned in association with captchas. I think disabled people have enough problems already, there's no reason to further annoy them with captchas (I'm even annoyed by them as a not-disabled person).

Although it's not about programming, I'd recommend "The design of everyday things" by Donald A. Norman. It gives a good idea on how to design user interfaces (including, but not limited to software). It's really well-written and nice to read, wether or not you know something about user interface design.