An important thing to realize is that multi-tasking is NOT the same thing as as an app/OS being multi-threaded. While most apps need to be specifically coded to be multi-threaded, operating systems for a long time have had the ability to take advantage of multiple CPUs to complete tasks.
Now, while a big jump in a single application may come from taking a huge CPU task and chopping it up into little pieces, there are definitely some tasks that lend themselves very easily to being multi-threaded. For example, probably the most important one is independently-executed Javascript threads. Browser performance can really be improved from multiple CPUs chewing on Javascript threads and then powering down to a low power state. Now, will it really matter when most of the wait for a page loading is downloading images? Probably not, but better performance is still better performance.
The key goal with dual/quad core chips is making sure the system itself still feels responsive when doing tasks. A good example of this is if you have an iPhone and you are listening to a video podcast while running Safari, the system will definitely see some slowdown. Or running any app while the OS is installing something from the App Store, uploading a photo to Flickr in the background, or streaming Pandora. As refined as smartphones seem, they are still just pocket computers with limited resources.