It's easier to filter out ambient conversations because you can hear all of the dialog. With cellphone conversations, however, you hear only one side. Your brain then works to fill in the missing dialog. Because your brain is working harder, cellphone conversations are more intrusive and annoying.
I suggest that you spend a fair amount of time in Japan, where public cellphone conversations are frowned upon (especially in trains). After coming back to the U.S., you'll realize how annoying cellphone chatters really are.