A game's systems work together to create an experience--in order for one to shine the others must be well designed, crafted, and integrated. Sometimes fixing a camera, tuning NPC stats, or adding some scripted chatter will do a lot more for the perceived intelligence of the opponent than a more sophisticated AI algorithm.
I think it's easy to lose sight of the ultimate goal (making a fun game) when coding AI, because, let's face it, making brilliant AI is a sexier challenge than paying attention to all the little details the make a level play well. Ironically, one of the problems in game AI these days is how to make the AI a little dumber (but not a lot dumber). It's easy to make a chess AI that will beat most people, but getting one's ass handed to one 30 times in a row is tiring to say the least. On the other hand, an AI that can be beaten handily isn't much fun either.
We don't really understand it, so we'll give it to the programmers.