If you break language into four tasks: speaking, listening, reading and writing, then speaking is by far the easiest.
I'd say that depends on what you consider "reading and writing". For Westerners, Mandarin is difficult to speak and listen to because of the tones--it takes a lot of practice for them to pronounce the tones properly, and also a lot of practice to distinguish the tones. Reading and writing is difficult because of the large number of characters that need to be memorized. However, if you're allowed to have computer assistance, reading and writing becomes much easier; I'd say easier than speaking and listening. You can easily look up a word in an online dictionary, and when typing, the IME will present you a list of possible characters, and you choose the one you want. The latter is a huge simplification, since you don't have to remember exactly how to write a character; you just need to have a general idea of what it looks like, and the IME will take care of the details. This is even affecting the current generation of Chinese people... it's not uncommon for even a college-educated person to draw a blank on how to hand-write a character: "Character Amnesia"