I was wondering the same thing too.
I have a decent 8-core xeon at work as my workstation. When I have intense computations to run, I do it on one of our clusters. The idea that someone would do intense calculations on a phone is pretty ridiculous. You've outlined a few good examples, but like you said, most of that is done with dedicated hardware, and seemingly instantly.
The only app that needs to be fast, and I mean really fast, is the app switcher and the Phone app. And they are pretty light on the computation anyway.