People are the best part of my job...and the worst. Technical challenges can be a real PITA but they don't even come close to comparing to the difficulty of people challenges. And of those challenges, communicating clearly with people is perhaps the most difficult part of my job on a daily basis. (The second is dealing with "problem people", who have some kind of personality issue that makes them difficult to work with.)
How many times have you heard someone start a sentence with "I thought what you meant was..." or "I thought you knew..." or something similar. Or said the same yourself. Communication is hard, and if you think otherwise you probably don't realize how many assumptions are being made on both sides when you talk to people.
The other hardest thing is getting a story (requirement) to "done done". Writing unit tests and getting them to pass is easy. The hard part is making sure I update the design document, project wiki, and deployment doc; that I check the license agreement for the library/framework I just downloaded; that I don't have SQL injection problems; that I've added system tests;